Hi everyone!

Winter has arrived, and so I had to check the thermal insulation of my ~~out of town residence~~ dacha. And it just turned out a famous Chinese marketplace started to sell cheap thermal camera modules. So I decided to DIY it up and build a rather exotic and useful thing — a heat visor for the home. Why not? Especially since I had a Raspberry Pi lying around anyway… The result is down below.

MLX90640. What is it?

This is a thermal camera matrix with an onboard microcontroller, made by an unknown (to me) company called Melexis. The matrix is 32x24 pixels, which isn’t a lot, but after interpolation it’s enough to notice general trends.

The sensor comes in two versions, the only difference being the case and the camera’s FoV. The more grounded A model observes the world with 110 degrees horizontally and 75 vertically. The B model has 55 and 37.5 degrees respectively. The case has four outputs — two for power and two for talking to a controller device via I2C. The datasheet can be found here.

What’s GY-MCU90640, then?

Our Chinese fellows also ship the MLX90640 chip with another microcontroller on board (STM32F103), probably for easier matrix control. The whole unit is called GY-MCU90640, and it costed me around 5000 RUB (roughly $80) in December 2018. It looks like this:

As we can see, there are also two versions of this model, with different sensors

Which one will work best? Unfortunately, I only asked myself this question after the module has been ordered, shipped and received. I haven’t thought about it when choosing.

The wider-angle version is best suited for self-driving robots or CCTV systems (since its field of view is better). The datasheet says it’s also less noisy and more accurate.

But for visualization I’d recommend the more “eagle-eyed” B model, for one very important reason. It can be turned in place (manually or via a drive) to make combined images way more detailed than its 32x24 resolution. But I don’t have one, so later on I’ll be talking about the wider-angled A model.

Connecting to the Raspberry Pi

We can control the thermal camera in two ways:

Short the “SET” pins on the board and use the I2C protocol to control the MLX90640 microcontroller directly
Leave the pins be and use the STM32F103 controller through RS-232 or similar interface.

If you code in C++, you’re probably better off ignoring the extra controller, short the pins and use the manufacturer’s API, found here.

Humble Pythonists could also use the first option. There appears to be a couple of Python libraries (here and here), but neither worked out of the box for me.

Advanced Pythonists could theoretically write their own controller driver. The datasheet explains how to extract a frame out of it. But you’ll have to describe all the calibration procedures manually, which I find excessively hard. So I used option 2. It turned out to be a bit convoluted, but still manageable.

Thanks to Chinese ingenuity (or luck), the output configuration on the board turned out to be very convenient:

All I needed to do was to insert the board into Raspberry’s port. The board has a 5V-3V converter built in, so the delicate Rx and Tx outputs of the Pi aren’t in any danger.

I’d also add that you could connect it similarly while using option 1, but you’ll have to be extremely careful and proficient in soldering. The board has to be mounted on the other side of the Pi (example is in the header photo).

Software

The famous Chinese marketplace offers this majestic piece of software for accessing the GY-MCU90640:

Apparently there also has to be some description of the communication protocol used to access the microcontroller, and after a short chat with the seller (big respect to him), I had said protocol in my hands. In PDF and in pure, distilled Chinese.

Thanks to Google Translate and a healthy dose of copy-pasting, around 90 minutes later the protocol has been decoded. I uploaded it on

GitHub. Turned out that the board understand 6 basic commands, including one for requesting the current frame via a COM port.

Every pixel of the matrix is essentially a reading of the object’s temperature. The temperature value is in degrees Celsius multiplied by 100 (a 2-byte number). There’s even a special mode when the board sends frames to the Pi automatically 4 times per second.

The full script for receiving thermal images:

"""MIT License

Copyright (c) 2019 

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE."""

import serial, time
import datetime as dt
import numpy as np
import cv2

# function to get Emissivity from MCU
def get_emissivity():
	ser.write(serial.to_bytes([0xA5,0x55,0x01,0xFB]))
	read = ser.read(4)
	return read[2]/100

# function to get temperatures from MCU (Celsius degrees x 100)
def get_temp_array(d):

	# getting ambient temperature
	T_a = (int(d[1540]) + int(d[1541])*256)/100

	# getting raw array of pixels temperature
	raw_data = d[4:1540]
	T_array = np.frombuffer(raw_data, dtype=np.int16)
	
	return T_a, T_array

# function to convert temperatures to pixels on image
def td_to_image(f):
	norm = np.uint8((f/100 - Tmin)*255/(Tmax-Tmin))
	norm.shape = (24,32)
	return norm

########################### Main cycle #################################
# Color map range
Tmax = 40
Tmin = 20

print ('Configuring Serial port')
ser = serial.Serial ('/dev/serial0')
ser.baudrate = 115200

# set frequency of module to 4 Hz
ser.write(serial.to_bytes([0xA5,0x25,0x01,0xCB]))
time.sleep(0.1)

# Starting automatic data colection
ser.write(serial.to_bytes([0xA5,0x35,0x02,0xDC]))
t0 = time.time()

try:
	while True:
		# waiting for data frame
		data = ser.read(1544)
		
		# The data is ready, let's handle it!
		Ta, temp_array = get_temp_array(data)
		ta_img = td_to_image(temp_array)
		
		# Image processing
		img = cv2.applyColorMap(ta_img, cv2.COLORMAP_JET)
		img = cv2.resize(img, (320,240), interpolation = cv2.INTER_CUBIC)
		img = cv2.flip(img, 1)
		
		text = 'Tmin = {:+.1f} Tmax = {:+.1f} FPS = {:.2f}'.format(temp_array.min()/100, temp_array.max()/100, 1/(time.time() - t0))
		cv2.putText(img, text, (5, 15), cv2.FONT_HERSHEY_SIMPLEX, 0.45, (0, 0, 0), 1)
		cv2.imshow('Output', img)
		
		# if 's' is pressed - saving of picture
		key = cv2.waitKey(1) & 0xFF
		if key == ord("s"):
			fname = 'pic_' + dt.datetime.now().strftime('%Y-%m-%d_%H-%M-%S') + '.jpg'
			cv2.imwrite(fname, img)
			print('Saving image ', fname)
		
		t0 = time.time()

except KeyboardInterrupt:
	# to terminate the cycle
	ser.write(serial.to_bytes([0xA5,0x35,0x01,0xDB]))
	ser.close()
	cv2.destroyAllWindows()
	print(' Stopped')

# just in case 
ser.close()
cv2.destroyAllWindows()

Results

The script polls the thermal matrix and outputs frames to the connected monitor’s console, 4 times per second, which is enough to not experience too much discomfort. For visualization it uses the OpenCV package. When you press S, “heat maps” from the camera are uploaded as JPGs to the script’s folder.

For better viewability I also made the app display minimum and maximum temperature within the frame. So, by looking at the heat map we can estimate the temperature of the coldest and hottest objects (within a degree, usually on the higher side), within the range of 20-40 degrees. Ctrl+C exits the script.

The script works the same on the Raspberry Pi Zero W and Pi 3 B+. I installed a VNC server on my smartphone, so, by carrying a Pi connected to a power bank with a VNC-enabled smartphone we can get a pocket thermal camera that saves images. Might not be too convenient, but it does the job.

After the first boot it might display the maximum temperature incorrectly, in which case just rebooting the script should do the job.

That’s about it for today. The experiment could be considered a success. You can definitely do a thermal scan of a house using this device. If someone can come up with other uses for this, please write in the comments.

Happy work week and see you soon!

UPD: I was asked in the comments to take a shot of the house from the outside. Here it is. The pictures ended up not very informative due to lower contrast of temperatures. The two upper photos are the entire house from two angles. The two lower photos are different windows.

The only change I made to the code was the temperature range: from +20...+40 to -10…+5.

Making a DIY thermal camera based on a Raspberry Pi