Initial Thoughts/Questions

[c-trimm]

Hey Dan! Just some thoughts/questions I had while looking over the project:

Code

1. Any interest in using Typescript? If so, I've got a template project I use for my own projects that I could adapt this code to pretty quickly. It's based on an ejected create-react-app, so the scripts are a little heavy-handed, but solid. It supports TS for both client/server, and is already set up to serve both concurrently with hot-reloading, and for deployment to Heroku (granted you probably don't want that part, haha). Might need some tweaks to work in docker, but that shouldn't be too hard.

2. Any interest in using a component library like MaterialUI or ReactBootstrap?

3. I think I know your thoughts on eslint/prettier, but figured I'd ask if there's any interest in using them for consistency between devs?

4. I noticed it seems like you're planning to roll your own auth. Curious on your thoughts of using a third party like Okta for user management to potentially speed up the MVP? Granted, that introduces potential security and data management risks.

5. I have an urge to comment on some of the documentation to ask questions directly, but GH doesn't provide a good way to do this. Any thoughts on using google docs or notion or something for it?

6. What's the plan (if any yet) for database migrations?

Features

1. I noticed that the primary way for users to submit papers is to upload them. I'm assuming this is because most papers are going to be in PDF format and to avoid users from having to copy/paste/format things into a RTE or form of some sort? I was thinking that if papers are in text-format, users could easily comment inline on things like they do during GH reviews. Maybe something like https://mathpix.com/ocr could be used to convert an uploaded PDF, and then let the user tweak as needed before submission. Could also be a good way to provide diffs for updates and "PRs".

2. I was curious about the peer review / reputation idea outlined in the peer-review planning doc -- it seems to be suggesting that only people with a high enough reputation (in a given field?) can peer review docs. Just an thought/idea I had about this - would be interesting if anyone could potentially comment on a doc, but if their reputation was low, their comments would be below users with higher reputation and perhaps dimmed to indicate the user has a low rep. This allows anyone to potentially point out interesting points or flaws in a paper, while gaining reputation if others upvote their comments. Spam could be an issue, though. Maybe handled through moderation and collapsing of comments that are severely downvoted?

Higher-level Project Questions

These might be easier to chat about out loud rather than here, but I figured I'd ask questions here while they're on my mind.

1. Source Credibility -- I'm asking this one with some admitted ignorance of the academic peer review process. I believe that one of the benefits of journals is that papers submitted are vetted to some degree, and so if you're linking to a paper from specific journals, readers can quickly assess if the paper is going to be a certain quality or not. Perhaps I'm mistaken on this, but if I'm not, do you foresee any issues with the credibility of papers being referenced from this site? I imagine reputation could help here, but I'd have some concerns about users gaming the system. I.e. psuedo-scientific users upvoting low quality or misleading papers/comments so that certain people could gain a high reputation, even though their papers aren't scientifically sound; making it seem like the author has a high reputation when they really shouldn't.

[danielBingham]

Hey Casey!

I'll take the questions in turn :)

Code

1. Eventually, yes, but not at first. Typescript is an initial cost and you only really get the benefits once you have a reasonably sized team. I don't want to spend extra time right now writing Types and creating type harnesses for libraries when I've only got a few months of runway. When it's just me (and at most a few volunteers) working on the code - and the code base is pretty small - it's not too difficult to stay on top of what object is what. If it gains traction and I get to build a team around it, then I think a Typescript migration would definitely be on the table at some point.

2. Maybe? Depends on the library and the cost. I'll take a look at those two (any other suggestions?)

3. Similar to typescript. I don't think the benefits outweigh the time it would take to set them up for now. Those are things that become a net benefit once you have a team. I thought about eslint, and I might do that in July if I have a little spare runway.

4. I'm trying to keep costs as low as humanly possible to keep the runway as long as possible. I think I can roll a reasonably secure and complete auth in a pretty limited timeframe, and then I don't have to pay monthly costs. Honestly, I think rolling Okta would probably take me a similar amount of time. I've done auth systems before. They're finicky, but if you don't over do it, they aren't too bad. (Edit: Looking at Okta's pricing, there's no way I could afford it right now. I don't think I could even afford Auth0.)

5. Not against it. For now I'm keeping documentation with the repo for simplicity's sake. Kinda trying to keep everything in Github for now, because it makes it all a little easier to find (and avoids tooling cost). Keeps things simple. We can achieve some level of commenting using PRs, but since it's pretty much just me right now (with a little help from my brother) I haven't been worrying about it as much and have just be doing whatever's easiest for me.

6. I'll cross that bridge when I come to it. For now I'm tunnel visioned on a releasable MVP (limited runway and all that).

Features

1. Most academics currently submit papers to journals as either PDF or docx. They do their writing in the tool of their choice (which is Word or an equivalent for a lot of them) and then submit in the journal's requested format. I'm trying to keep the submission process as familiar as possible (while also making it easy). My hope is that the MVP would support PDF, Docx, and latex. Docx and latex can both be convert to HTML to allow in-line commenting. For PDFs I'm using pdf.js - which will allow me to access the PDF's text content, but also renders it to a canvas. So the way it works right now is that you can click on the canvas and the plan is to tie the comment to that canvas location and show it off the side the way Google Docs does. Not exactly the same as Github reviews, but I think it's good enough UX. With time we could potentially figure out how to do inline even for PDFs. Although, the way most papers are formatted, I'm not sure that's actually desireable. Off to the side may actually be better. Go download a paper from an Open Access journal (like PLOS One) as a PDF and try uploading it to Peer Review. You can see the rough alpha of what I have in mind (right now the comments just hover where ever you click and are tied to page location - I didn't want to deal with the fiddly math yet and wanted something I could show that gives the gist to get feedback).

2. So, we can talk about this in more detail later, but the TL;DR is that review is just about editorial feedback for the author. The reputation system comes into play more once the paper is published and public. Users with enough reputation can offer reviews pre-publish, and then vote up or down and offer public responses post publish.

Okay, longer explanation: Right now, peer review in the journals serves two purposes: editoral feedback (helping authors polish their work) and quality control (keeping bad or dishonest work from getting published). The latter has stopped functioning. When a paper gets rejected from a reputable journal, a dishonest author can just take it to a pay to play journal that doesn't have quality checks. As far as the public is concerned, the end result is the same - it got published. There are over 10,000 journals, even academics are having a hard time keeping straight which ones are reputable. And it's changed - some open access journals that were promising early on have dropped in quality, meaning they've become a net negative on the CVs of academics who published with them before their quality dropped.

My intention is to take peer review and split its two functions. Review on a draft will solely be about polishing the work and giving editorial feedback. The quality control will come from the reputation system post-publish.

Here's how it works: An academic posts a draft and tags it with the relevant fields. Users who have enough reputation in those fields can offer reviews of the draft. The author can accept or reject the reivews based on whether or not they are helpful (granting reputation to those accepted). When the author is ready, they publish the paper. At that point, users who have enough reputation in those fields can vote it up or down (granting the author reputation) and post public responses (which themselves can be voted up or down by others in the fields with reputation and granting less reputation).

Tieing reputation to the fields and restricting those features to reputation in those fields is necessary (at least at first) in order to get buy-in. Academics don't want to have to sift through a ton of feedback from people with out expertise in their fields. They wouldn't use the site.

Eventually, if it gets traction, my thought was to introduce comments on public papers (separate from the responses) that anyone could post to ask questions, make suggestions, and generally interact with the experts in the field. In that way, the site could facilitate scientific communication. But I don't think that's an MVP feature.

Higher-level Project Questions

1. Yeah, see above - this is where the reputation system tied to the fields comes in. And the plan is to seed it by using email/name to look up an existing citation metric (maybe H-score, maybe total citations, maybe i10) and grant starting reputation using that. But also, yeah, getting that credibility is going to be the most difficult part about gaining traction. I'm hoping that's where the file drawer problem, the reputation system, and my connections into academia (plus the readiness of everyone for change) will help. But that's also why the great likelihood is that I'm going to be back on the job market in 4 - 5 months 🤷‍♂️

[c-trimm]

Thanks for the extremely thorough response, Dan. All of that makes complete sense to me, especially with the time frame you've got. I like how you're going about the reputation stuff, makes a lot more sense than what I was imagining!

With all of that said, is there anything I can do to help things along?

Also, is the code in main up to date? I had to make some changes to get things running (e.g. the logging in the server was broken), and when I uploaded a paper, it doesn't actually let me view the upload or anything (from what I can tell). Either I'm doing something wrong, missing something, or you've got some stuff locally that's not pushed.

[danielBingham]

Cheers! To view a paper after upload you may need to refresh the page and then you should be able to see it on the "submissions" page (as yet unsquashed bug). And you should be able to make review comments on your own paper, I don't have any kind of checking for that yet. But you can also just register as many arbitrary emails as you want - there's no confirmation. I've been using [email protected] and [email protected] for testing along with a certain paper titled "Molecular Structure of Nucleic Acids" ;)

I got most of what you see written during a week of vacation, and I was pushing as far as I could towards a skeleton I could show to my academic friends to give them the gist of what I have in mind, so it's all very "Happy path only". The only thing on main locally are some unit tests which I haven't pushed yet - actually I'll go push them. I pushed some documentation updates last night and just added a few more (the beginnings of an API spec) tonight.

As for helping... give me a few weeks (okay 6 weeks...) and then definitely!! At the moment, it's all a little tangled and in my head and not in a good place for parallelization. I think with in a week or two of my exit I'll be able to have the skeleton of the MVP done and then spend a little time to define the work needed to get to a releasable MVP and ways in which it could be parallelized. I can get them organized into issues and the project and at that point, I would welcome help!

[danielBingham]

Okay, turns out I broke a bunch of stuff when I started adding Fields and introduced the logger to the backend (stuff the unit tests didn't catch - wee!). Should be fixed now. So you should be able to submit a paper and be taken directly to the submission review page. Plus, you should be able to tag it with fields now!