Play Video With ESP32

This Instructables show something about playing video and audio with ESP32.

Step 1: ESP32 Features & Limitations

Features

• 4 SPI bus, 2 SPI bus available for user space, they are SPI2 and SPI3 or called HSPI and VSPI. Both SPI buses can run at most 80 MHz. Theoretically it can push 320x240 16-bit color pixels to SPI LCD at 60 fps, but it has not yet counted the time overhead required for read and decode the video data.
• 1-bit / 4-bit SD bus can connect SD card in native protocol
• I2S internal DAC audio output
• over 100 KB RAM available for video and audio buffer
• Fair enough processing power to decode JPEG (play Motion JPEG) and LZW data compression (play Animated GIF)
• Dual-core version can split read data from SD card, decode and push to SPI LCD into parallel multi-tasks and boost the playback performance

Limitations

• not enough internal RAM to have double frame buffer for 320x240 in 16-bit color, it limited the multitask design. It can overcome a bit with external PSRAM though it is slower than internal RAM
• not enough processing power to decode mp4 video
• not all ESP32 version have 2 core, the multi-task sample only benefit on dual-core version

Ref.: https://en.wikipedia.org/wiki/ESP32

Step 2: Video Format

RGB565

Or called 16-bit color is a raw data format commonly used on the communication between MCU and color display. Each color pixel represented by a 16-bit value, the first 5-bit is red value, following 6-bit is green value and then 5-bit blue value. 16-bit value can make 65536 color variation so it also called 64K colors. So 1 minute 320x240@30 fps video will be sized: 16 * 320 * 240 * 30 * 60 = 2211840000 bits = 276480000 bytes or over 260 MB

Animated GIF

This is a common file format on the web since 1990s. It limit the color variation for each screen up to 256 colors and do not repeat store the pixel that as same color as previous frame. So it can much reduce the file size, especially when each animation frame not change too much details. The LZW compression is designed capable decoded by 1990s computer, so ESP32 also have fair enough processing power to decode it in real time.

Motion JPEG

Or called M-JPEG / MJPEG is a common video compression format for the video capture hardware with limited processing power. It actually simply a concatenation of still JPEG frames. Compare with MPEG or MP4, Motion JPEG no need computationally intensive technique of interframe prediction, every frame is independent. So it requirement less resource to encode and decode.

Ref.:

https://en.wikipedia.org/wiki/List_of_monochrome_a...

https://en.wikipedia.org/wiki/GIF

https://en.wikipedia.org/wiki/Motion_JPEG

Step 3: Audio Format

PCM

A raw data format for digital audio. ESP32 DAC use 16-bit bit depth, that means each 16-bit data represent a digital sampled analog signal. Most video and song audio commonly use sample rate at 44100 MHz, that means 44100 sampled analog signal for each second. So, 1 minute mono audio PCM raw data will be sized: 16 * 44100 * 60 = 42336000 bits = 5292000 bytes or over 5 MB. The size of stereo audio will be double, i.e. over 10 MB

MP3

MPEG Layer 3 is a compressed audio format widely used for song compression since 1990s. It can dramatically reduce file size to under one-tenth of raw PCM format

Ref.:

https://en.wikipedia.org/wiki/Pulse-code_modulatio...

https://en.wikipedia.org/wiki/MP3

Step 4: Format Conversion

This project use FFmpeg convert the video into ESP32 readable format.

Please download and install FFmpeg at their official site if not yet: https://ffmpeg.org

Convert to PCM audio

ffmpeg -i input.mp4 -f u16be -acodec pcm_u16le -ar 44100 -ac 1 44100_u16le.pcm

Convert to MP3 audio

ffmpeg -i input.mp4 -ar 44100 -ac 1 -q:a 9 44100.mp3

Convert to RGB565

ffmpeg -i input.mp4 -vf "fps=9,scale=-1:176:flags=lanczos,crop=220:in_h:(in_w-220)/2:0" -c:v rawvideo -pix_fmt rgb565be 220_9fps.rgb

Convert to Animated GIF

ffmpeg -i input.mp4 -vf "fps=15,scale=-1:176:flags=lanczos,crop=220:in_h:(in_w-220)/2:0,split[s0][s1];[s0]palettegen[p];[s1][p]paletteuse" -loop -1 220_15fps.gif

Convert to Motion JPEG

ffmpeg -i input.mp4 -vf "fps=30,scale=-1:176:flags=lanczos,crop=220:in_h:(in_w-220)/2:0" -q:v 9 220_30fps.mjpeg

Note:

• FFmpeg converted Animated GIF can be further optimizered by some web tools, you may search GIF optimizer to find one.

Step 5: Hardware Preparation

ESP32 Dev Board

Any dual-core ESP32 dev board should be ok, this time I am using a TTGO ESP32-Micro.

Color Display

Any color display that Arduino_GFX support should be ok, this time I am using a ILI9225 breakout board with SD card slot.

You can find Arduino_GFX supported color display list at Github:

https://github.com/moononournation/Arduino_GFX

SD Card

Any SD card should be ok, this time I am using a SanDisk "normal speed" 8 GB micro SD with SD adaptor.

Audio

If you want to use headphone only, simply connect headphone pins to pin 26 and GND can listen the audio. Or you can use a tiny amplifier to play audio with speaker.

Others

Some breadboards and breadboard wires

Step 6: SD Interface

ILI9225 LCD breakout board also included a SD crd slot breakout pins. It can be used as SPI bus or 1-bit SD bus. As mentioned in my previous instructables, I prefer using 1-bit SD bus, so this project will base on 1-bit SD bus.

Step 7: Put It Together

The above pictures show the testing platform I am using in this project. The white breadboard is 3D printed, you can download and print it at thingiverse: https://www.thingiverse.com/thing:4552162

The actual connection depends on which hardware you have in hand.

Here are the connection summary:

ESP32
Vcc     -> LCD Vcc
GND     -> LCD GND
GPIO 2  -> SD D0/MISO -> 1k resistor -> Vcc
GPIO 14 -> SD CLK
GPIO 15 -> SD CMD/MOSI
GPIO 18 -> LCD SCK
GPIO 19 -> LCD MISO
GPIO 22 -> LCD LED
GPIO 23 -> LCD MOSI
GPIO 27 -> LCD DC/RS
GPIO 33 -> LCD RST

Ref.: https://www.instructables.com/id/Dev-Board-Breadb...

Step 8: Program

Arduino IDE

Download and install Arduino IDE if you are not yet do it:

https://www.arduino.cc/en/main/software

ESP32 Support

Follow the Installation Instructions to add ESP32 support if you re not yet do it:

https://github.com/espressif/arduino-esp32

Arduino_GFX Library

Download latest Arduino_GFX libraries: (press "Clone or Download" -> "Download ZIP")

https://github.com/moononournation/Arduino_GFX

Import libraries in Arduino IDE. (Arduino IDE "Sketch" Menu -> "Include Library" -> "Add .ZIP Library" -> select downloaded ZIP file)

ESP8266Audio

Download latest ESP8266Audio libraries: (press "Clone or Download" -> "Download ZIP")

https://github.com/earlephilhower/ESP8266Audio

Import libraries in Arduino IDE. (Arduino IDE "Sketch" Menu -> "Include Library" -> "Add .ZIP Library" -> select downloaded ZIP file)

RGB565_video Sample Code

Download latest RGB565_video sample code: (press "Clone or Download" -> "Download ZIP")

https://github.com/moononournation/RGB565_video

SD Card Data

Copy the converted files to SD card and insert into LCD card slot

Compile & Upload

1. Open SDMMC_MJPEG_video_PCM_audio_dualSPI_multitask.ino in Arduino IDE
2. If you are not using ILI9225, change the new class code (around line 35) to correct class name
3. Press Arduino IDE "Upload" button
4. If you failed to upload the program, try detach the connection between ESP32 GPIO 2 and SD D0/MISO
5. If you find the orientation not correct, change the "rotation" value (0-3) in new class code
6. If program run well you can try other sample start with SDMMC_*
7. If you do not have SD card slot or you don't have FFmpeg installed, you can still try SPIFFS_* example

Step 9: Benchmark

Here are the performance summary for different video (220x176) and audio (44100 MHz) format:

Format	Frame per second(fps)
MJPEG + PCM	30
GIF + PCM	15
RGB565 + PCM	9
MJPEG + MP3	24

Note:

• MJPEG + PCM can reach higher fps but it is unnecessary play in a tiny screen greater than 30 fps
• RGB565 does not require decode process but the data size is too large and much time consumed at loading data from SD, 4-bit SD bus and faster SD card can improve it a little bit (wild guess can reach around 12 fps)
• MP3 decode process not yet optimized, it is now dedicate core 0 for MP3 decode and core 1 for playing video

Step 10: Happy Playing!

Now you can play video and audio with your ESP32, it unlocked many possibilities!

I think I will make a tiny vintage TV later...

Comments

[jujumanstudios]

Hi community,
How are you? I hope you are doing well despite this pandemic.
Thanks 陳亮 for posting this instructable.
I need help with a little project.
I just started a course in Arduino I'm a complete newbie.
I saw your tutorial on play video with esp32.
I build plastic kits e.g. robots (Gundams), aircraft, etc.
I'd like to install a 0.96" lcd creen (or similar size) to play videos using Arduino or other microcontrollers.
The link below is what I want to achieve.
https://www.youtube.com/watch?v=69zOxR1Z-Hs&t=132s
On the video above it seems that he presses a button to change the video.
Do you think is possible? Usually, the LCD of this size doesn't have an sd card reader, so I don't have any idea on how to do this.
Anyone up for tutoring?
Please let me know.
Thanks
Claudio (Juju)

[vividhobby]

Dear Author
Thank you for your advice on this movie project.
Please lallow me to report something regarding my trial on this.
(Result)
The Best FR could be 15FPS with 44100 Audio through
SD Para4 and / or SD HSPI(16MHz) to get acceptable IQ
and sound Quality.
(Software)
My target is to get faster FR with audio through SD.
I have been trying to get LCD Para8 $ SD Para4 by modification of
"SPIFFS_MJPEG_JPEGDEC_MP3_audio_ESP32I2S8".
However I gave it up because I could not solve error regarding
SD_MMS on Audio process.
Then I tried to modify "SD_MJPEG_video_PCM_audio" to get
both of LCD Para8 and SD Para4.
In addition, I compare to the performance between SD Para4 and
SD HSPI.
(Detail)
When I increase FR from 15 to 20, Image looks better, but playback
speed seems slower than sound.
Image and sound are NOT synchronized.
P.S. If you have some advice to increase FR more with this setting,
I appreciate it.
Regards
VividHobby

[陳亮]

I think 15 fps is already very good enough for 320x240 resolution. There is no meaning step up to 20 or 24 fps except it can reach the normal standard, 30 fps.

[vividhobby]

Thank you for advice.
I tried to set more than 20MHz clock on SPI , but it was limited below 20MHz by SD library. If I could exceed this, its FR could be increased a littel bit more. Could you plase advise how I can change its library ?
I appreciate it .

Regards
VividHobby

[陳亮]

still 4-bit SDMMC is fastest

[vividhobby]

Dear Author

Thank you for making an effort to this Video Project on your web.
Finally, after many cut & try, I got

"SPIFFS_MJPEG_JPEGDEC_MP3_audio_ESP32I2S8.ino"

working on mu ILI9341. (refer image-1 attached)
Followings are some changes from your original program.
I am not sure if my modification was correct, but I appreciate it if you correct me
when I was wrong.

1) in the program "SPIFFS_MJPEG_JPEGDEC_MP3_audio_ESP32I2S8.ino"
"FFat.begin()" is used, but I switched it to "SD_MMC.begin()" because
I could NOT observe any SD card Read/Write signal on my LogisAnalizer.
After this change, SD data is Read/Write succesfully.
I am not sure if this change was correct. (refer image-2 attached)

2) When I try to view your demo video, it was clearly displayed on my screen.
However some frams were skipped.
Program reported, 18% frames were skipped.(refer image-3 attached)
Is this usual result ?
Thank you !

Regards
VividHobby

[陳亮]

FFat is a new filesystem library for replacing SPIFFS, it can run faster than SPIFFS, i.e. much faster than SD card. Currently only FFat can reach 30 fps.

[vividhobby]

Thank you for information. Yes, I was aware of that after asking my question.
It is great to achieve FR30. Only limitation is size of SPIFFS.(< 3GB)
My recommendation could be Parallel LCD & 4bit SD. FR is around 25, but no sound avairable due to Pin# limitation (as Iong as know). DA Audio pins (25 / 26) are already occupid. umm .....

[陳亮]

you can change PAR8 pins that

[vividhobby]

Thank you for good advice.
I tried to find I/O pins for LCD(para8) and SD(para4), but unfortunately I couldn't find spare I/O pins except DA(25, 26) below pin# 33.
Can I use pin#34, 35 for I/O ?

[陳亮]

if you are using the ESP32 without PSRAM, uyou can use those pins, e.g. 12, 13 and 14.

[vividhobby]

Dear Arthur
Thank you very much for immediate reply.
My ESP32 is " ESP32WROOM-32D ".
Pin#12, 13 and 14 are already assigned for SD as follows.
All (I/O ports) is occupied already except DA audio(25, 26) and exceed 34 as follows.
This means there is no FREE pint to be assigned.
Is there any idea to have another I/O for LCD D5 pin to replace "Pin# 25 DA" ?
If I can use Pin# 35, 35 , it will be great. But I don't know how.


SD Card | ESP32
==============
D2 12
D3 13
CMD 15
VSS GND
VDD 3.3V
CLK 14
VSS GND
D0 2 (add 1K pull up after flashing)
D1 4

LCD ESP32
==============
Reset 33
DC 27
CS 5
WR 17
RD 32
D0 23
D1 19
D2 18
D3 22
D4 21
D5 25-->> conflict with DA audio (25, 26)
D6 0
D7 16

Regards
Vivid

[陳亮]

Oh yes, 12, 13 and 14 is SD pins. The PSRAM pins should be 16 and 17, I found you already used it. You can always pull-up the LCD RD pin and free up GPIO 32 for LCD DC pin. Then you can use GPIO 27 for LCD D5. And also you can always pull-down the LCD CS pin to free up GPIO 5.

[vividhobby]

Dear Arthur
Thank you for advice regarding Pin# assignment.
I tried it, and got it working on LCD and SD.

Original setting for AUDIO is as follows.
"aFile = new AudioFileSourceFS(SD_MMC, MP3_FILENAME);"
But
Is it correct if I switch it from FS to SD (para 4)
that "aFile = new AudioFileSourceSD(SD_MMC, MP3_FILENAME);"
But I got some error.

Any advice for SD(para4) audio play back ?
If you advise some example, it will be great for me to study.

Regards
Vivid

[陳亮]

no need to change, it already SD_MMC

[vividhobby]

Dear Author

Thank you for your advice for me to achieve "Movie / sound Display Project".

I got it working of PLAYING Video only with setting as follows.
----------------------------------------------
File vFile = SD_MMC.open(MJPEG_FILENAME);
aFile = new AudioFileSourceFS(FFat, MP3_FILENAME);
-----------------------------------------------
even it Audio is set as FFat.
Of cause NO sound. But Video is working good.
Next, I set Audio as follows.
------------------------------------------------------
aFile = new AudioFileSourceFS(SD_MMC, MP3_FILENAME)
------------------------------------------------------
With this setting, there was NO video and No audio.
It looks crashed in the SETUP().
After displaying "PCM audio MJPEG video start", the system is Aborted,
and then self Reset.
Is there any points I have to chack ?
Thank you for your advice.
Thank you !
Regards
VividHobby
------ Log -------------------------------
ets Jun 8 2016 00:22:57
rst:0x1 (POWERON_RESET),boot:0x1b (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:1
load:0x3fff0018,len:4
load:0x3fff001c,len:1216
ho 0 tail 12 room 4
load:0x40078000,len:10944
load:0x40080400,len:6388
entry 0x400806b4
E (168) psram: PSRAM ID read error: 0xffffffff
PCM audio MJPEG video start
#0 draw buffer allocated
#1 draw buffer allocated
#2 draw buffer allocated
#3 draw buffer allocated
drawTask start
E (9752) sdmmc_req: sdmmc_host_wait_for_event returned 0x107
E (9753) sdmmc_cmd: sdmmc_read_sectors_dma: sdmmc_send_cmd returned 0x107
E (9753) diskio_sdmmc: sdmmc_read_blocks failed (263)
Skip frame
ets Jun 8 2016 00:22:57
**** Self Reboot ****************
rst:0x8 (TG1WDT_SYS_RESET),boot:0x1b (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:1
load:0x3fff0018,len:4
load:0x3fff001c,len:1216
ho 0 tail 12 room 4
load:0x40078000,len:10944
load:0x40080400,len:6388
entry 0x400806b4
.......

[陳亮]

try set PSRAM: disable

[vividhobby]

Hello
Thank you very much for your kind advices.
I tried it as you mentioned, but not succesfull unfortunately with followings.
Rebuild new video with 10FPS, and Audio (22050.mps) and disable PSRAM.
My expectation on this parallel settings for LCD and SD is "higher FR" a little bit lower than SPIFFS. (maybe 20-25 FPS)
Actually, I could achieve 15 FPS with Para8_LCD and SPI_SD. and 25 FPS
with Para8_LCD and Para4_SD without Audio. My expectation was about 20FPS with Audio.
What can I do ?
Thank you !!

Regards
Vivid

[陳亮]

you may try reduce FPS, say 10, to test is it work.

[vividhobby]

Dear Author
I would like to re-address tweaking on Arduino Para8.

I may have found the root cause of that issue.(hopefully)
Please have a look at the attached three pictures showing the Logic Timing of "WR" and "Data" signals of "UNO", "LovyanGFX" and "ArduinoGFX".
Let's check the "Falling Edge" and "Rising Edge" of RW and Data pulse in both of "UNO" and "Lovyan" image.
Their "Rising Edge" is clearly observed within the data pulse period. But in the ArduinoGFX case, "Rising Edge" is observed at the same timing of "Rising edge" of the Data pulse.
This is important, bucase LCD reads data / command when "WR" rising timing.
If the rising edge of "WR" is very close to data edges, the LCD may have ERR to read correctly.
What do you think about this possibility ? and any idea to avoid this timing error ?

VividHobby