Thanks for sharing! I had a few questions I'd really appreciate if you could answer.
1. How do you envision this process gets implemented within the current neuroscience pipeline? Will it be a tool for current researchers to make them more effective? Will it be self-sufficient?
2. There is a massive regulatory barrier in drug development—there are too many proposed drugs than feasible number of drug trials to run. Assuming we automate this ideation phase, we'd likely be overwhelmed by new hypotheses, how would we manage the influx of new ideas for later experimental phases?
3. (This is more methodology related) How do you test the "efficacy" of this tool? What training dataset would exist to make this work?
4. My current understanding of LLMs is that they work extraordinarily well on data that already exists in excess but is poor in novel thought, because there's less training data. In working at the frontier of neuroscience, how effectively does this agent create these unique, original ideas?
Very very interested and excited by the work, and I hope to follow along as this project further develops!
These are some very thoughtful questions! I'm glad you enjoyed learning about my work. Here are my answers:
1. There is currently an open-source tool used by many computational neuroscientists where you can perform computational "experiments" on the fruit fly brain called FlyBrainLab(https://github.com/FlyBrainLab/FlyBrainLab) I will be integrating the AI theory validation within this existing tool, and later integrating with other tools that are used by computational scientists. I am a pretty big fan of going "with the grain" when it comes to new tools and starting with integrations in the tools that the target user already uses.
2. You're absolutely right about the regulatory barrier. Neuroscience operates like a funnel. Computational testing sits at the wide top, where initial ideas are vetted before the best candidates progress to experimental stages, then to costly clinical trials, with ultimately only about 6% of central nervous system trials receiving FDA approval.
If we automate the ideation and computational testing phases, computational neuroscientists could dramatically increase front-end filtering before ideas reach more expensive and time-intensive stages. Currently, a typical computational neuroscience paper contains just 3-4 computational "experiments." By streamlining this process, we could feasibly see papers featuring 20-30 computational experiments, allowing significantly more poor concepts to be identified and eliminated at this early, low-cost stage.
The efficiency gains could be substantial, filtering out even a few additional percentage points of flawed ideas during the ideation/computational stage could potentially double the success rate of downstream clinical trials. This means more effective treatments reaching patients with the same research budget and timeline constraints.
3. I will be testing the efficacy of the tool by doing usability testing with neuroscientists! Usability testing of technology intended to help professionals is a well-developed research area with robust standardized methodologies. As part of my larger dissertation proposal I have plans to have real neuroscientist attempt to use the system to help them with real research projects, and to do usability testing to see whether the system succeeds, and what its' strong and weak points are. As far as training data, in early testing that we've done, foundation LLMs are actually quite capable of everything that we need them for. I don't expect to be doing any fine-tuning or needing any training dataset.
4. You are absolutely correct! The earliest version of the system had the LLM creating the theories as well, but they weren't good enough at doing any kind of original or novel thinking to come up with any really interesting or useful theories. For the things the system is actually doing, I'm actually not having the LLM do anything terribly novel at any point. I have a highly multi-agent architecture where the task any individual agent has to do at any step is fairly simple and something that LLMs are known to excel at. An example is that I might have one LLM who has access to a small set of papers, and I ask it "please summarize the types of theories that could be tested with the methods in these papers". I then pass this along to another LLM along with the theory and say "Based on this list of the types of theories this method could be used for, and this theory, tell me whether or not I could use the method to test this theory", I keep going down the list with more and more agents coordinating with each other. Each individual step is something that most high school-educated adults could accomplish with almost no training. However, if I have a team of 100 of these agents then I can combine them to do much higher-level work.
Thanks for the thoughtful questions and I am glad to see that someone out there is excited about my project!
Thanks for sharing! I had a few questions I'd really appreciate if you could answer.
1. How do you envision this process gets implemented within the current neuroscience pipeline? Will it be a tool for current researchers to make them more effective? Will it be self-sufficient?
2. There is a massive regulatory barrier in drug development—there are too many proposed drugs than feasible number of drug trials to run. Assuming we automate this ideation phase, we'd likely be overwhelmed by new hypotheses, how would we manage the influx of new ideas for later experimental phases?
3. (This is more methodology related) How do you test the "efficacy" of this tool? What training dataset would exist to make this work?
4. My current understanding of LLMs is that they work extraordinarily well on data that already exists in excess but is poor in novel thought, because there's less training data. In working at the frontier of neuroscience, how effectively does this agent create these unique, original ideas?
Very very interested and excited by the work, and I hope to follow along as this project further develops!
These are some very thoughtful questions! I'm glad you enjoyed learning about my work. Here are my answers:
1. There is currently an open-source tool used by many computational neuroscientists where you can perform computational "experiments" on the fruit fly brain called FlyBrainLab(https://github.com/FlyBrainLab/FlyBrainLab) I will be integrating the AI theory validation within this existing tool, and later integrating with other tools that are used by computational scientists. I am a pretty big fan of going "with the grain" when it comes to new tools and starting with integrations in the tools that the target user already uses.
2. You're absolutely right about the regulatory barrier. Neuroscience operates like a funnel. Computational testing sits at the wide top, where initial ideas are vetted before the best candidates progress to experimental stages, then to costly clinical trials, with ultimately only about 6% of central nervous system trials receiving FDA approval.
If we automate the ideation and computational testing phases, computational neuroscientists could dramatically increase front-end filtering before ideas reach more expensive and time-intensive stages. Currently, a typical computational neuroscience paper contains just 3-4 computational "experiments." By streamlining this process, we could feasibly see papers featuring 20-30 computational experiments, allowing significantly more poor concepts to be identified and eliminated at this early, low-cost stage.
The efficiency gains could be substantial, filtering out even a few additional percentage points of flawed ideas during the ideation/computational stage could potentially double the success rate of downstream clinical trials. This means more effective treatments reaching patients with the same research budget and timeline constraints.
3. I will be testing the efficacy of the tool by doing usability testing with neuroscientists! Usability testing of technology intended to help professionals is a well-developed research area with robust standardized methodologies. As part of my larger dissertation proposal I have plans to have real neuroscientist attempt to use the system to help them with real research projects, and to do usability testing to see whether the system succeeds, and what its' strong and weak points are. As far as training data, in early testing that we've done, foundation LLMs are actually quite capable of everything that we need them for. I don't expect to be doing any fine-tuning or needing any training dataset.
4. You are absolutely correct! The earliest version of the system had the LLM creating the theories as well, but they weren't good enough at doing any kind of original or novel thinking to come up with any really interesting or useful theories. For the things the system is actually doing, I'm actually not having the LLM do anything terribly novel at any point. I have a highly multi-agent architecture where the task any individual agent has to do at any step is fairly simple and something that LLMs are known to excel at. An example is that I might have one LLM who has access to a small set of papers, and I ask it "please summarize the types of theories that could be tested with the methods in these papers". I then pass this along to another LLM along with the theory and say "Based on this list of the types of theories this method could be used for, and this theory, tell me whether or not I could use the method to test this theory", I keep going down the list with more and more agents coordinating with each other. Each individual step is something that most high school-educated adults could accomplish with almost no training. However, if I have a team of 100 of these agents then I can combine them to do much higher-level work.
Thanks for the thoughtful questions and I am glad to see that someone out there is excited about my project!