Implement Trending Packages into Backend

[confused-Techie]

Per popular request, this PR finally implements a "Trending" feature into the backend.

Since currently we only have a simple "download" count per package, there's never been any way to track when a package is downloaded (or even if those downloads happened pre-Pulsar). But this PR finally implements what we need to do just that.

This PR creates a new table, that tracks the download download of each package. Creating an entry for a package per day as it's download. Allowing us an aggregate count of how many times a day a package is downloaded.

Which we can then in turn use SQL to get the total download count over any arbitrary time interval.

Although, this PR just contains some of the first steps, a more complete look of what we need to do between this PR and having this feature live looks like:

• Add API endpoints to retrieve trending counts
• Add tests for new API endpoint, and new DB behavior
• Ideally, add caching mechanism for these counts, as a query per user can be expensive
• Add new microservice to automatically clean up this new DB over time, so we don't retain information forever.
• Implement this new API endpoint into the packages website
• Implement this new API endpoint into settings-view

Some key questions we may want to consider now that this is being worked on:

• Many services track trending across the download/uninstall ratio. We already have a no-op endpoint, inherited from Atom, of when a user uninstalls a package. Should that now update a packages download trend?
• With a proper trending implementation, should we discontinue the previous "Featured Packages"? We rarely updated it as is, and I'd imagine this would replace it's spot on the packages website, but what do we think?

[Daeraxa]

Love to see it, some notes:

• Personally, yes, I'd like to see stats about packages being uninstalled. Even if we don't expose it or use it for a trending page it could provide useful insight or early warning to pre-emptively investigate broken packages based on stats alone. However whilst I don't actually see it as an issue, is there a chance people might see this as collecting anonymous usage stats? Obviously there is an easy workaround to just remove the package from your config by hand but just throwing it out there.
• I think the featured page is worth retiring, yes. I like the idea that we can hand pick and promote packages but I don't think we have the bandwidth to keep doing it and it becomes stale which looks bad from both user and package maintainer perspectives. The trending page should replace it entirely.
• I think we should be able to use more fields for "fancier" trending algorithms or "weighting". "Trending" should not just mean "popularity over time period" (but is fine to start with). Right now (now as in this PR) we just have "daily" table that records the ID of the package, the day, and how many downloads? I don't know if we would need to add to this table or if it is fine to keep this and join the original tables in but if we had:
  • Created data: We could weight a "trending" package as higher based on how recent the package was published. For example if 1000 people download linter and 500 download my-new-package then I would argue the latter indicates it is "trending" much more than the former.
  • Deltas: If we see that linter gets 10000 downloads in week1, 9000 in week 2, 11000 in week 3 then the delta is low, this isn't "trending" it is just "popular". But if my-new-package gets 10 in week 1, 50 in week 2, 500 in week 3 then the delta is much larger and again more deserving of "trending"
  • Metadata: Using the metadata allows us to use something like keywords to define categories. This means we could curate the trending page by limiting or reducing the rank of packages that share categories and promote those in different categories. This also extends to theme vs "normal" packages.
  • Package dependencies: If a package is automatically installing another package then there is justification to rank the dependency lower. For example if nearly all of the ide type packages depend on linter but people aren't actually searching for and installing linter by itself then it isn't something we would want to show as trending.

I'm sure there are a ton of other cases but these were just some quick notes I thought worth adding.

[savetheclocktower]

I do think that it would count as telemetry if we had ppm hit an endpoint whenever the user uninstalled a package, since it isn't at all necessary from a technical perspective. I doubt it would bother many people… but I think it'd be something worth disclosing, if only so that users could opt out of it.

[savetheclocktower]

Also: in the long-term, the “featured” list is something we probably don't have the bandwidth to maintain. But until such time as we retire it, we might as well swap in some packages that are more likely to be among the most common packages a user would want to install. Maintaining such a list is a high-effort task overall, but changing the current fixed list of featured packages is a low-effort task.

[confused-Techie]

Thanks a ton for your thoughts at @Daeraxa & @savetheclocktower!

So for the point about telemetry. The uninstall endpoint was inherited from Atom. We still have it, we just don't ever do anything with it. So if we worry that this counts as telemetry, then we can continue to do absolutely nothing with it (the endpoint just immediately returns success to keep PPM happy).

As for calculating trending, you make some wonderful points on how we should determine what trending means, and the cool thing, is with these changes we still can.

All we have right now is a table that tracks downloads of a package per day. That's the basis for any of your other suggestions.

So we could currently say any of the following without much hassle:

• Hey this package got the most downloads this week, that means it's trending: Probably the simplest way to do this, but still somewhat valid.
• Hey this package had 5 downloads last week and 100 this week, it's trending: More akin to what you are suggesting, and can still be done with just two DB queries instead of one.
• Hey this package got half it's total download count last week, it's trending: Again more complex, and may require several queries but still possible.

The only point, if we are wanting to do this more complex calculation of trending, we likely won't be able to do it in real time. We'd likely want to run a scheduled task that does these calculations then saves the results somewhere. Maybe only updating every week or something, especially since anything other than "lots of recent downloads" as trending would require more than one somewhat expensive query.

But I do love the ideas you're coming up with here, and I think they'd be a bit more valuable than what I had in mind.
As for instances of depended packages becoming trending, I don't think there's any good way to really automate that. That might be something that we just have to discover over time, and manually exclude that package from trending or something. Or see if there's some way we can tell by user agent how the package was downloaded.