---

This is the repository for CodeScientist, an end-to-end semi-automated scientific discovery system that designs, iterates, and analyzes scientific experiments that can be expressed as (Python) code. CodeScientist creates novel ideas to explore essentially by using genetic mutations (using an LLM-as-a-mutator paradigm) to mutate combinations of scientific articles and code examples, with code examples including how to prompt an LLM, make a plot, or use a specific benchmark. The experiment ideas can then be implemented using the Experiment Builder, which automatically creates, runs, and debugs the experiment code in a container. When completed, CodeScientist writes a report on the results. Usually, CodeScientist makes several (for example, 5) independent attempts at creating experiments for a given idea, and can create a meta-analysis describing the overall results over each of the 5 experiment attempts.

CodeScientist can be run in two modes: - Human-in-the-loop: A human helps build code examples, filter experiment ideas to run, and provides short comments on the ideas that might help their implementation. This is the primary mode we report in the paper. - Fully-automatic: You can run CodeScientist in fully automatic mode with a few clicks, though it is less efficient at producing scientific results.

What you'll find in this repository: - CodeScientist Software: CodeScientist is open source, and this repository includes the full set of software and installation instructions. - Reports: The CodeScientist paper highlights a set of 20 candidate discoveries (in Table 4). These are readily available here: Example CodeScientist-Generated Experiment Reports and Code

• Raw Data: The repository also includes a great deal of raw data: full experiment code, logs, ideas, external reviewer ratings, etc.

---

Table of Contents

• 0. Paper
• 1. Quick Start
• 1.1. I want to read about CodeScientist
• 1.2. I want to examine the papers, code, and other results created by CodeScientist
• 1.3. I want to run CodeScientist on my local machine
• 1.4. I would like to use CodeScientist in my own domain
• 1.5. I want to manually provide CodeScientist an idea to create an experiment for
• 1.6. I want to feed ideas into CodeScientist from another system
• 1.7. How do I use a specific aspect of CodeScientist?
• 1.8. I have a question not answered here
• 2. Example CodeScientist Generated Experiment Reports and Code
• (Experiment 1 – Experiment 6)
• 3. Installation and Running
• 3.1. Installation
  • Cloning the repository
  • Modal Setup
  • API Key Setup
  • LLM Proxy Setup
• 3.2. Running the Server and GUI
• 4. Using CodeScientist
• 4.1. Create New Ideas (from Papers)
• 4.2. Create New Experiment (Manual)
• 4.3. Ideation List
• 4.4. Batch Autonomous Experiments
• 4.5. Run Benchmark
• 4.6. Experiment Monitor
• 4.7. Bulk Reporting and Meta-Analysis
  • Generating New Reports
  • Viewing Reports
• 4.8. Pregenerated Ideas/Filtering Ideas Externally
  • Step 1: Run the batch ideator
  • Step 2: Run batch idea simplifier and ranker
  • Step 3: Filter down the ideas
  • Step 4: Make plans for the (filtered) ideas
• 4.9. Your First Experiment -- "Hello World"
• 5. Adding Codeblocks to the Codeblock Library
• 5.1. Browsing existing codeblocks
• 5.2. Adding a new codeblock
• 6. The Experiment Builder Sandbox
• 6.1. Container Parameters
• 6.2. LLM Proxy
• 6.3. Example LLM usage log
• 6.4. Example Debugger Log (log.json)
• 6.5. LLM-generated code breaking the sandbox
• 7. Data
• 7.1. Included Data
• 7.2. Data Not Included in the Repository
• 7.3. Resetting the Data
• 7.4. Additional Data Available
• 8. Benchmark
• 9. Using parts of CodeScientist in your own codebase
• 10. Prompts
• 11. Frequently Asked Questions
• 12. Danger Zone
• 13. Citation
• 14. License
• 15. Contact

0. Paper

CodeScientist is described in the following paper: CodeScientist: End-to-End Semi-Automated Scientific Discovery with Code-based Experimentation.

1. Quick Start

1.1. I want to read about CodeScientist

The CodeScientist paper is available here: Section 0. Paper

1.2. I want to examine the papers, code, and other results created by CodeScientist

• Appendix: A number of the highest quality experimental results (as rated by humans) are in the paper's Appendix.
• Example Papers: You can also see the above highly rated experimental results (and a number of rejected papers) here: example_papers/
• Lots of Papers: If you'd like all the details -- high-quality and low-quality experiments, including their papers, code, results, and logs, they are available in bulk here: generated_experments/

1.3. I want to run CodeScientist on my local machine

Please see the installation instructions in Section 3.1. Installation

1.4. I would like to use CodeScientist in my own domain.

To use CodeScientist in a subdomain other than the provided domain (i.e. agents and environments), there are two steps: - Add papers in the subdomain: This is as easy as pasting Arxiv links into the Create New Ideas (from Papers) menu item. - Add codeblocks: If you need specialized codeblocks for your domain other than the general ones provided in this repository, simply add them to the codeblocks directory in the required format.

More information on these steps is provided in Section 3. Installation and Running and Section 4. Using CodeScientist

1.5. I want to manually provide CodeScientist an idea to create an experiment for, instead of using LLM-generated ideas.

You can do this by pressing the Create New Experiment (Manual) button on the main menu. More detailed instructions on running CodeScientist are provided in Section 4.2. Create New Experiment (Manual)

1.6. I want to feed ideas into CodeScientist that were made from some other system.

You can do this in bulk using the Run Benchmark button -- see the section on pre-generating ideas for an example of the format CodeScientist expects here: Secton 4.8 .Pregenerated Ideas/Filtering Ideas Externally, followed by Section 4.5. Run Benchmark

1.7. How do I use [specific aspect of CodeScientist]?

Please see the instructions for various components in the detailed usage instructions of Section 4. Usage Instructions

1.8. I have a question not answered here.

Please see the documentation below. If you're question isn't answered, please add an issue.

2. Example CodeScientist Generated Experiment Reports and Code

Below are six example experiment reports (and supporting code) generated by CodeScientist (from Table 4 in the CodeScientist paper). Note that the full experiment logs, code, results files, etc., for these (and other) experiments are available in: example_experiments/

(The reports are intended to be quickly-read overviews that describe the bulk of the experiment, and do not include literature reviews or discussions)

Experiment 1: Analyzing LLM Confidence in TextWorldExpress State Predictions

Summary: This paper examines the relationship between large language model (LLM) confidence scores and prediction accuracy in a text-based game environment. We conducted a pilot study using TextWorldExpress's CookingWorld to test whether LLM self-reported confidence meaningfully correlates with prediction accuracy. Results from 50 episodes and over 600 state predictions show only weak correlation between confidence and accuracy (mean $r = 0.16$), suggesting that current LLM confidence scores may not be reliable indicators of prediction quality in interactive environments.

📄 Paper PDF | 💻 Code

Experiment 2: Impact of State Representation Complexity on LLM Simulation Accuracy in CookingWorld

Summary: This paper investigates how increasing state representation complexity affects the ability of large language models (LLMs) to accurately simulate state transitions in the CookingWorld environment. We tested four levels of state complexity (boolean, numerical, relational, and full) and measured prediction accuracy across 25 episodes with up to 25 steps each. Our results show a clear inverse relationship between state complexity and simulation accuracy, with boolean representations achieving the highest accuracy (94.5\%) and full state representations the lowest (81.9\%). These findings suggest that while LLMs can effectively simulate simple state transitions, their performance degrades significantly with increased state complexity.

📄 Report PDF | 💻 Code

Experiment 3: Comparing Single-Stage vs Two-Stage Approaches for Text Game Generation

Summary: This paper investigates whether a two-stage approach to generating text-based games using large language models (LLMs) produces more complete and robust implementations compared to single-stage generation. We conducted a controlled experiment comparing single-stage generation against a two-stage process where basic mechanics are generated first, followed by scoring and win conditions. Results show that while both approaches achieved 100\% execution success, the two-stage approach produced significantly more complete game implementations (96.7\% vs 66.7\% mechanics completion rate), though at the cost of longer generation times. These findings suggest that decomposing complex game generation tasks into focused subtasks leads to higher quality output.

📄 Report PDF | 💻 Code

Experiment 4: Evaluation of LLM-Based vs Mathematical Approaches for Resistor Substitution

Summary: This paper evaluates three approaches for suggesting resistor combinations to match target resistance values: an LLM-based advisor, a simple baseline using nearest values, and a mathematical optimization approach. Testing on 50 random target values between 10$\Omega$ and 1M$\Omega$ showed that while the LLM approach achieved only 24\% accuracy within 1\% tolera