added dynamic sitemap.xml and robots.txt SEO routes in seo.routes.js

[PrinceK-Git]

Added two new Express routes for SEO:

• /sitemap.xml — Dynamically queries MongoDB and generates an XML sitemap
  listing all course pages. Auto-updates when new courses are added.
• /robots.txt — Tells search engine bots which pages to crawl (course pages)
  and which to skip (dashboard, profile, API, admin).

Files changed:

• NEW: server/modules/seo/seo.routes.js
• MODIFIED: server/index.js (imported and registered the new routes)

Part of Issue #190 (SEO Implementation)

[DreamBot706]

Few critical issues found in this pr:

---

1. Critical: URL Encoding for Course Codes with Spaces

File: server/modules/seo/seo.routes.js (line 33)

<loc>${BASE_URL}/browse/${course.code}</loc>

Some courses in CourseHub have spaces in their codes (e.g., "CH 222"). According to the sitemaps.org protocol and RFC 3986, raw space characters are strictly prohibited inside <loc> tags. If GSC encounters a space, it will fail validation with:

"Invalid XML: URL contains whitespace"

Fix:
Strip whitespace or URL-encode the course code:

<loc>${BASE_URL}/browse/${encodeURIComponent(course.code.replace(/\s+/g, ""))}</loc>

---

2. Express Middleware Order in server/index.js

File: server/index.js (line 87)

app.use((error, req, res, next) => {
    // global error handler...
});
app.use(seoRoutes); // <-- Mounted AFTER error handler!
app.get("*", (req, res) => ...);

In Express, error-handling middleware must always be mounted last (after all route definitions). Because seoRoutes is currently mounted after the error handler, any unhandled database error in seo.routes.js will bypass your central error logger.

Fix:
Move app.use(seoRoutes); up alongside the other API route declarations (e.g., right below line 64: app.use("/api/student", studentRoutes);).

---

3. Defensive Date Parsing in sitemap.xml

File: server/modules/seo/seo.routes.js (lines 27–29)

const lastmod = course.updatedAt
    ? course.updatedAt.toISOString().split("T")[0]
    : new Date().toISOString().split("T")[0];

Because .lean() returns plain JavaScript objects, if any course in MongoDB has updatedAt stored as an ISO string or undefined, calling .toISOString() directly will throw TypeError: course.updatedAt.toISOString is not a function and cause a 500 error.

Fix:
Wrap safely in new Date(...):

const lastmod = course.updatedAt && !isNaN(new Date(course.updatedAt))
    ? new Date(course.updatedAt).toISOString().split("T")[0]
    : new Date().toISOString().split("T")[0];

---

4. robots.txt Path Coverage & HTTP Caching

1. Path Coverage: In robots.txt, adding /browse alongside /browse/ ensures crawlers landing on https://coursehub.codingclub.in/browse (without trailing slash) are explicitly matched:

   Allow: /
   Allow: /browse
   Allow: /browse/
   

2. Cache Headers: Adding res.set("Cache-Control", "public, max-age=86400, s-maxage=86400"); to /sitemap.xml prevents repetitive MongoDB queries when search bots re-crawl the sitemap.

---

[DreamBot706]

1. Remove docs/nginx.conf from this PR

Action: Delete docs/nginx.conf from this branch.

Why:
The docs/nginx.conf you guys added had a bunch of issues and was not consistent with the correct one you are using and dont keep the nginx.conf in the repo.

Don't worry: I've already updated the Nginx files corresponding to your pr.

---

2. JavaScript .replace() Dollar Sign ($) Bug

File: server/middleware/seo.js (lines 91–99)

let modified = html.replace(
    /<title>CourseHub<\/title>/i,
    `<title>${title}</title>`,
);

modified = modified.replace(
    /<meta\s+name="description"[^>]*\/?>/i,
    injectedBlock,
);

Why this breaks:
In JavaScript, String.prototype.replace(regex, string) treats the $ character in the replacement string as a special token ($1 = regex capture group, $& = match, etc.). If any course name or JSON-LD field contains a dollar sign or math symbol ($), JavaScript will treat it as a token and silently delete or corrupt all characters following it.

Fix: Pass a replacer function instead of a raw string:

let modified = html.replace(
    /<title[^>]*>[\s\S]*?<\/title>/i,
    () => `<title>${title}</title>`,
);

modified = modified.replace(
    /<meta\s+name="description"[^>]*\/?>/i,
    () => injectedBlock,
);

---

3. Browser Tab Titles for Humans (CourseHub & CS101 | CourseHub)

Remember the separation of concerns:

• Crawlers/Bots need the long, keyword-stuffed title for Google indexing (CS101 - Introduction to Computing Study Materials | CourseHub IIT Guwahati). Express middleware handles this.
• Human Students need clean, short, readable tab titles in their browser.

A. Landing Page (client/src/screens/landing/index.jsx):

The title currently set ("CourseHub | IIT Guwahati Study Materials, Notes & PYQs") is way too long for a browser tab and gets cut off. For humans, keep it clean and simple as just CourseHub. Also, please fix the indentation in that useEffect block.

In client/src/screens/landing/index.jsx:

useEffect(() => {
    document.title = "CourseHub";
}, []);

B. Course Browse Page (client/src/screens/browse/index.jsx):

Right now, browse/index.jsx doesn't update document.title at all — when a student browses a course, the tab just stays generic. When viewing a course, the tab should update to ${courseCode} | CourseHub (e.g. CS101 | CourseHub).

In client/src/screens/browse/index.jsx, add:

useEffect(() => {
    if (code) {
        document.title = `${code.toUpperCase()} | CourseHub`;
    } else {
        document.title = "CourseHub";
    }

    return () => {
        document.title = "CourseHub";
    };
}, [code]);

---

4. Google Search Console URL Inspection Tool is Bypassing the Middleware

File: server/middleware/seo.js (lines 26–27)

BOT_UA_REGEX currently only looks for googlebot. However, when you go into Google Search Console and click "Test Live URL" or "Inspect URL", Google uses:

Mozilla/5.0 ... (compatible; Google-InspectionTool/1.0;)

googlebot does not match Google-InspectionTool. GSC will be treated as a human, bypass your middleware, and report that the page has no metadata!

Fix: Update BOT_UA_REGEX to include GSC's tool, Lighthouse, and missing crawlers:

const BOT_UA_REGEX =
    /googlebot|google-inspectiontool|chrome-lighthouse|storebot-google|bingbot|slurp|duckduckbot|baiduspider|yandexbot|facebot|facebookexternalhit|twitterbot|discordbot|whatsapp|telegrambot|linkedinbot|slackbot|applebot|ia_archiver|msnbot|ahrefsbot|semrushbot|dotbot|rogerbot|360spider|sogou|bytespider/i;

---

5. In-Memory Query Caching on SEO Middleware (Performance & DB Protection)

File: server/middleware/seo.js (lines 116–127)

Right now, every bot request to /browse/<code> fires an unindexed $regex query in MongoDB (flexPattern). Course codes change at most once per semester. If an aggressive crawler or scraper hits hundreds of course links, these regex queries will create unnecessary database load.

Fix: Add a simple in-memory cache (a Map with a timestamp TTL or simple object cache) for course code lookups:

const courseCache = new Map(); // key: normalizedCode -> { course, expiresAt }
const CACHE_TTL_MS = 60 * 60 * 1000; // 1 hour

Check the cache before querying MongoDB, and store the result.

---

6. Missing og-image.png Asset

File: server/middleware/seo.js (line 60)

const ogImage = `${BASE_URL}/og-image.png`;

og-image.png doesn't actually exist in the repo. When links are shared on WhatsApp or Discord, the crawler asks for https://coursehub.codingclub.in/og-image.png and gets a 404, so link preview thumbnails fail.

Fix: Create/export a 1200×630 banner and place it at client/public/og-image.png so Vite copies it to the web root during build.

---

7. Quick Polish & Defensive Coding

1. Add Canonical Link: In server/middleware/seo.js inside injectedBlock, add:

   <link rel="canonical" href="${courseUrl}" />

   This prevents Google from penalizing duplicate URLs (/browse/cs101 vs /browse/CS101).

2. Vary Header on 404: In server/middleware/seo.js (line 130), before returning res.status(404).send(...), add:

   res.set("Vary", "User-Agent");

   Otherwise, upstream caches like Cloudflare might cache the 404 HTML and serve it to humans.

3. Cache-Control for robots.txt: In server/modules/seo/seo.routes.js (line 66), add:

   res.set("Cache-Control", "public, max-age=86400, s-maxage=86400");

   Matches the 24-hour cache header already on sitemap.xml so crawlers don't repeatedly hit Express.

4. Consolidate BASE_URL: In server/middleware/seo.js and server/modules/seo/seo.routes.js, BASE_URL is hardcoded. Use config.clientURL from server/config/default.js (or process.env.CLIENT_URL || "https://coursehub.codingclub.in").

5. Defensive Check in Sitemap: In server/modules/seo/seo.routes.js (line 26), add:

   for (const course of courses) {
       if (!course?.code) continue;
       ...

   If any legacy course record in MongoDB has a null/empty code, course.code.replace() will throw a TypeError and crash the entire /sitemap.xml request.

6. cachedIndexHtml Refresh: In server/middleware/seo.js, cachedIndexHtml is cached indefinitely in memory. If a frontend-only deploy runs without a backend PM2 reload, Express will serve stale asset references to bots. Give it a simple TTL (e.g. re-read every 10 minutes) or check file mtime.

---