Modular ESP32 controller for robotic flutes and simple wind instruments.
Servo Flute GMB converts a recorder, tin whistle, ocarina, Native American flute, transverse flute, bansuri, shakuhachi, ney, kaval, or similar instrument into a MIDI-controlled robotic instrument.
Finger count, servo channels, fingering tables, airflow behavior, MIDI settings, and calibration values are configured at runtime from the embedded web interface. Recompiling the firmware is not required for normal instrument setup.
Important
Project status: the firmware builds and its software behavior is covered by automated tests. Physical validation is still in progress. Hardware-dependent features remain marked NOT TESTED — requires hardware until verified on the corresponding ESP32, PCA9685, servos, valves, pumps, fans, sensors, microphone, and instrument.
Warning
This project controls mechanical and pneumatic actuators. Use a separate fused actuator supply, a common ground, proper flyback protection for inductive loads, and a physical emergency-stop or power-disconnect method. Operate the web interface only on a trusted network.
- Up to 31 finger servos using one or two PCA9685 boards
- Runtime-configurable finger count, PCA channels, servo direction, closed angle, and half-hole position
- Instrument presets and fully editable MIDI fingering tables
- Six modular air-management modes
- BLE-MIDI, rtpMIDI/AppleMIDI, serial MIDI DIN, virtual keyboard, and local MIDI-file playback
- Automatic General-Midi-Boop recognition and capability reporting, generated from the active configuration
- Embedded responsive web interface for configuration, tests, monitoring, and calibration
- Optional INMP441 microphone auto-calibration
- Per-note minimum, nominal, and maximum airflow values
- Persistent JSON configuration stored in LittleFS
- Safe boot ordering, centralized configuration validation, watchdog handling, actuator-test timeouts, and panic behavior
This firmware targets wind instruments whose pitch can be controlled mainly by closing holes and adjusting airflow:
- recorder and tin whistle;
- Native American flute;
- ocarina and tabor pipe;
- transverse flute, bansuri, dizi, and fife;
- shakuhachi, ney, kaval, and similar end-blown flutes.
Reed and lip-buzzing instruments such as clarinet, saxophone, trumpet, and trombone are outside the current scope because they require additional embouchure control.
flowchart LR
BLE[BLE-MIDI] --> WM[Selected wireless mode]
RTP[rtpMIDI / AppleMIDI] --> WM
DIN[Serial MIDI DIN] --> IM[InstrumentManager]
FILE[Local MIDI file] --> IM
WEB[Web keyboard] --> IM
WM --> IM
IM --> SEQ[NoteSequencer]
SEQ --> FINGERS[FingerController]
SEQ --> AIR[Air controllers]
FINGERS --> PCA[PCA9685 #1 / #2]
PCA --> SERVOS[Finger servos]
AIR --> FLOW[Flow / angle / valve servos]
AIR --> GPIO[Solenoid / fan / pumps]
MIC[INMP441 microphone] --> CAL[Auto-calibration]
SENSORS[ToF / Hall / endstops] --> AIR
CAL --> IM
The physical switch selects BLE mode or Wi-Fi mode. Serial MIDI can remain available independently when enabled. BLE-MIDI and rtpMIDI are therefore not both active through the selected wireless mode at the same time.
Start with the smallest reproducible configuration before adding pumps, reservoirs, sensors, or automatic calibration.
| Component | Quantity | Purpose |
|---|---|---|
| ESP32-WROOM development board | 1 | Main controller |
| PCA9685 | 1 | Servo PWM controller |
| SG90-class servos | 7 | Six fingers and one airflow servo |
| Regulated 5 V servo supply | 1 | Separate actuator power |
| Bulk capacitor near PCA9685 V+ | 1 | Limits voltage dips caused by servos |
| Air valve or airflow mechanism | 1 | Starts and stops the note |
| Logic-level MOSFET driver | 1 | Required for a solenoid or DC load |
| Flyback diode | 1 | Required across an inductive load |
| Fuse or resettable protection | 1 | Actuator-supply protection |
| BLE/Wi-Fi selector switch | 1 | Connected to GPIO4 |
| INMP441 microphone | Optional | Automatic pitch/airflow calibration |
- Do not power multiple servos from the ESP32 5 V pin.
- Power the PCA9685 servo rail from a dedicated regulated supply.
- Connect ESP32 ground, PCA9685 ground, servo-supply ground, and driver ground together.
- Size the supply for servo startup and stall current, not only average current.
- Add a fuse and a physical way to remove actuator power.
- Use a MOSFET and flyback diode for solenoids, pumps, relays, and other inductive loads.
| Function | ESP32 pin / bus |
|---|---|
| Status LED | GPIO2 |
| BOOT / pairing button | GPIO0 |
| BLE/Wi-Fi selector | GPIO4 |
| PCA9685 output enable | GPIO5 |
| I2C SDA | GPIO21 |
| I2C SCL | GPIO22 |
| Default solenoid output | GPIO13 |
| INMP441 BCLK | GPIO14 |
| INMP441 WS / LRCLK | GPIO15 |
| INMP441 data | GPIO32 |
The default PCA9685 addresses are:
- board 1:
0x40, global channels0-15; - board 2:
0x41, global channels16-31.
The second board is required only when the configuration uses channel 16 or above. Solder the appropriate address jumper on the second PCA9685 before connecting it.
Detailed pin and parameter information: Configuration and PCA9685 expansion.
| Mode | Architecture | Typical use |
|---|---|---|
| 0 | Solenoid + flow servo | Simple compressed-air or blower source |
| 1 | Servo valve + flow servo | Fully servo-operated valve system |
| 2 | Flow servo only | Minimal mechanism where the servo also stops airflow |
| 3 | Fan + flow servo | Continuous low-pressure blower |
| 4 | One to three pumps + valve | Direct pump supply without reservoir |
| 5 | Pumps + reservoir + valve | Regulated stored-air system with sensor feedback |
The web interface displays only the controls relevant to the selected mode. See Air management.
| Input | Availability | Typical source |
|---|---|---|
| BLE-MIDI | Bluetooth switch position | Phone, tablet, computer |
| rtpMIDI / AppleMIDI | Wi-Fi switch position | DAW on the local network |
| Serial MIDI DIN | Optional, independently enabled | Hardware MIDI controller |
| Web keyboard | Wi-Fi mode | Browser |
| MIDI file | Wi-Fi mode, stored in LittleFS | Autonomous playback |
All accepted events converge on the same monophonic InstrumentManager and NoteSequencer path so finger positioning, airflow timing, replacement notes, note-off handling, and panic behavior use the same logic.
The firmware implements the General-Midi-Boop instrument recognition and capability protocol, version 2. When the flute appears as a MIDI device, General-Midi-Boop probes it, reads a capability descriptor, and creates or updates the instrument entry by itself — no manual entry.
Everything announced comes from the active, validated configuration: the playable notes are built from the current fingering table, the MIDI channel, the announced control changes, and the timing model all follow the configuration in use. Nothing is hard-coded, and there is no second capability model in the firmware.
| Element | Value |
|---|---|
| Handshake | SysEx block 1, 24 bytes, protocol version 2 |
| Descriptor | JSON, served over SysEx block 0x10 in 200-byte segments |
| Change notification | SysEx block 0x11 after a validated and activated configuration change |
| HTTP descriptor | GET /gmb/descriptor.json (Wi-Fi mode) |
| Instance id | 32-bit, derived from the ESP32 eFuse MAC, stable across reboots |
| Transports | BLE-MIDI and rtpMIDI. Serial MIDI DIN is receive-only on this board and cannot be recognized automatically. |
Saving a configuration that really changes what the instrument can play increments a persistent revision counter, rebuilds the descriptor, and notifies General-Midi-Boop. A reboot alone never increments it.
A field the firmware has not measured is left out rather than guessed — an absent field means "unknown" in GMB. The acoustic excitation latency (timing.excite.latency_ms) is therefore not announced, and announcing 0 would tell General-Midi-Boop the instrument speaks instantly. The firmware now measures this latency — that is what the timing phase of the acoustic chain does — but the measurement only runs while the analysis is active, has never been checked against a real microphone or flute, and carries a bias that does not cancel. Announcing a figure that has been measured but not verified would be worse than announcing none. See Project status.
See General-Midi-Boop protocol.
The ESP32 embeds a responsive single-page application. No external server is required.
| Section | Function |
|---|---|
| Keyboard | Play configured notes and view hole and airflow states |
| MIDI | Upload, select, and play SMF Type 0/1 MIDI files |
| Air | Configure and monitor the selected pneumatic architecture |
| Calibration | Configure fingers, fingerings, breath, and expression |
| Settings | Instrument, MIDI, Wi-Fi, hardware, and persistent configuration |
The Air tab is shown only when relevant to the selected configuration.
REST endpoints and WebSocket messages are documented in Web API.
PlatformIO is the recommended build method because the repository already pins the target platform, partition layout, and library versions.
- Git
- Visual Studio Code with PlatformIO, or PlatformIO Core
- ESP32-WROOM-compatible development board
- USB data cable
git clone https://github.com/glloq/Servo-Flute-GMB.git
cd Servo-Flute-GMB
pio run
pio run --target upload
pio device monitorThe project uses a custom 4 MB partition layout because the complete firmware, embedded web application, BLE, Wi-Fi, MIDI, and calibration code exceed the default ESP32 application partition.
To run the host-side native tests:
pio test -e nativeThe exact dependencies and versions are listed in platformio.ini.
- Flash the firmware and restart the ESP32.
- Put the selector in Wi-Fi mode.
- On first boot, connect to the
ServoFlute-Setupaccess point. - Open the captive portal; if it does not appear automatically, open
192.168.4.1. - Select an instrument preset or define the finger count and embouchure type.
- Configure the PCA9685 channel used by each servo.
- Calibrate the closed position and direction of every finger.
- Configure or verify the fingering table.
- Select the air-management mode and test each actuator individually.
- Set per-note airflow values manually or use microphone auto-calibration.
- Save the configuration and restart when the interface reports
restart_required. - Connect a MIDI source and test at low actuator power before full operation.
When station credentials are saved, the interface is normally available through servo-flute.local or the IP address displayed by the device.
| Position | Active wireless mode |
|---|---|
| LOW | BLE-MIDI |
| HIGH | Wi-Fi, rtpMIDI, web interface |
| Action | Effect |
|---|---|
| Short press | Restart BLE advertising or display the Wi-Fi IP address |
| Double press within 500 ms | Open all fingers, unless an actuator session owns the hardware |
| Long press for 3 seconds | Force Wi-Fi access-point mode |
An INMP441 I2S microphone can measure the sounding result while the firmware sweeps airflow. For each note, the calibration system can determine:
- minimum usable airflow;
- recommended nominal airflow;
- maximum airflow before instability or overblow;
- confidence, tuning error, stability, and signal-to-noise information.
Beyond calibration, the same microphone feeds an acoustic analysis chain built in phases: ring-buffered acquisition with overlap, YIN pitch detection, Goertzel and optional FFT spectral analysis, stream filtering, a noise model per machine state (a pump at 90 % is not the same noise floor as a pump at rest, so one global floor would overrate every note played), acoustic classification with a quality score, and note timing — the latency between the order and the audible sound.
Read the validation level before trusting any of it. Nothing in this chain is above simulated audio validated: every figure comes from synthetic PCM, now passed through the production filter chain, and no INMP441 and no flute have ever been connected. The reference document defines five levels explicitly and marks each phase, and carries an audit section listing what is still wrong after correction — including up to +8.4 dB of optimism on a strongly low-frequency-weighted noise floor, which is exactly the pump case.
See Audio and acoustic architecture and Auto-calibration.
| Area | Status |
|---|---|
| ESP32 firmware build | Automated build available |
| Native behavior tests | Implemented |
| Embedded web application | Implemented |
| Configuration validation | Implemented |
| BLE / Wi-Fi / serial MIDI paths | Implemented in software |
| General-Midi-Boop recognition (blocks 1 / 0x10 / 0x11) | Implemented and unit-tested; wire format cross-checked against the current General-Midi-Boop parsers |
| Local MIDI-file playback | Implemented in software |
| Safe boot and actuator-test timeout logic | Implemented and regression-tested |
| Physical PCA9685 and servo validation | Requires hardware |
| Pump, fan, valve, and sensor validation | Requires hardware |
| INMP441 calibration validation | Requires hardware |
| Network authentication | Implemented (session tokens, generated admin password) |
Detailed audit findings and the physical test matrix are maintained in Project status.
The hotspot key and the web admin password are generated randomly at first
boot from the ESP32 hardware RNG, stored in NVS, and printed on the serial
console. Neither is derived from the MAC address. Every route that changes
something — configuration, reset, restart, filesystem recovery, MIDI files,
Wi-Fi — and every WebSocket command requires a session token obtained from
POST /api/auth/login; purely informative routes stay open.
Lost the password? Hold the BOOT button while the board powers up (5 s): both secrets are regenerated and printed on serial.
Authentication is not a substitute for the rest:
- there is no TLS, so the token travels in clear on the local link — the network is still the trust boundary;
- do not expose the ESP32 to the Internet or use port forwarding;
- disconnect actuator power when the system is unattended.
See Access model.
| Document | Description |
|---|---|
| Getting started / this README | Project overview and first build |
| Architecture | Firmware modules and data flow |
| Air management | Air modes, pumps, fans, reservoir, and sensors |
| Web API | REST endpoints and WebSocket protocol |
| Auto-calibration | INMP441 pitch and airflow calibration |
| Calibration | Manual instrument calibration workflow |
| Configuration | Runtime parameters and persistence |
| General-Midi-Boop protocol | Automatic recognition, descriptor, and capability reporting |
| PCA9685 expansion | Second board and global channel mapping |
| Wi-Fi modes | BLE selection, station mode, and access point |
| Serial MIDI | MIDI DIN input and optocoupler wiring |
| Servo mounting | Mechanical setup for transverse flutes |
| Project status | Validation status, limitations, and test references |
The following repository-native visuals should be added as real hardware and CAD material becomes available:
- complete instrument hero photograph or CAD render;
- minimal electrical wiring diagram;
- finger mechanism in open, half-open, and closed positions;
- six air-mode diagrams;
- screenshots of Keyboard, Calibration, and Air pages;
- short demonstration video or animated preview.
Recommended location: docs/assets/.
Servo-Flute-GMB/
├── README.md
├── LICENSE
├── platformio.ini
├── docs/
│ ├── ARCHITECTURE.md
│ ├── AIR_MANAGEMENT.md
│ ├── API_WEB.md
│ ├── AUTO_CALIBRATION.md
│ ├── CALIBRATION.md
│ ├── CONFIGURATION.md
│ ├── GMB_PROTOCOL.md
│ ├── PCA9685_EXPANSION.md
│ ├── SERVO_ANGLE.md
│ ├── STATUS.md
│ └── WIFI_MODES.md
├── Servo_flute_ESP32/
│ ├── Servo_flute_ESP32.ino
│ ├── settings.h
│ ├── gmb/ General-Midi-Boop recognition modules
│ └── firmware modules
└── tests/
Issues and pull requests should include:
- the instrument and air mode used;
- ESP32 board and PCA9685 count;
- relevant configuration values;
- exact reproduction steps;
- serial logs when applicable;
- whether the result was tested on physical hardware or only in software.
Do not report a physical test as passed unless it was executed on the corresponding hardware.
This project is licensed under the MIT License.