Week 25/08/2024 - 31/08/2024

26/08/2024

Took more pictures of object tools.

27/08/2024

Took more pictures of hands

Start annotating 1008 images for training and 207 images for validation.

28/08/2024

Wrote program to obtain coordinate and depth values of detected object.

The z value was measured with a measuring tape. The distance is the same distance as the camera (0.31m). 

Next, edit the program so that it identifies object mentioned by the user via text input.

Searched object: Extractor

Searched object: Scalpel

The next step is to coordinate the transformation from the camera's coordinate to the UR10's coordinate. Before that, the camera needs to be mounted onto the UR10. In order to do so, a mounting plate needs to be made. Measurements were obtained from the mechanical drawings of the Intel RealSense camera and the UR10 mounting bracket.

A mounting board was then designed using Inkscape for laser cutting. 

The two tutorials watched to create the svg, dfx and pdf file for the mounting board.

Design is done and ready to be sent to the laser cutter:

30/08/2024

Sent laser cut design to the workshop.

Finished product:

Finish setting up hardware: 

Attempt to perform camera - robot calibration using the checkerboard method, which didn't work by the end of the day. 

A checkerboard sheet was printed for calibration purpose. the idea was to sync the points of the checkerboard using the camera and the robot.

Checkerboard template was printed from here:

https://markhedleyjones.com/projects/calibration-checkerboard-collection

31/08/2024
Watched videos to gain more knowledge on robot-camera calibration:

Encountered issues with calibration. Robot arm moved way beyond the target...

Or bent at a very awkward angle:

Week 18/08/2024 - 24/08/2024

A few days were spent training the model because the accuracy level was not sufficient. Images for training was increased to 643 at various distances and angles and images for validation was increased to 131. 

When tested by holding the camera above the objects, the results were far from satisfactory. The detecting was fleeting at best, and not very accurate.

Even with a different background, the percentage of accuracy was not acceptable.

However, after mounting the camera onto a stable stand, the results seems to be much better, as shown in the video below:

As for the UR10, instead of coding the script on the laptop, the pendant was used to code the UR10 to move and pick up object from a fixed location. The URScript was then transferred from the pendant to laptop. This helped solved the issue of not being able to code the grippers to work. 

Meanwhile, data collection has begun for the object detection of the project's second part (hand / palm detection).

Week 11/08/2024 - 17/08/2024

Issue with object recognition model

After trouble shooting the programme for object recognition, the programme was still unable to detect the tools, whether through saved image or video. It was then discovered that the model's accuracy was very low and 30 epochs was not sufficient to build a good model.

After increasing the number of epochs to 50, the results were slightly better, but still not good enough.

Image shows two labels when there is only one tool (scalpel).

Inaccurate - unable to detect object (extractor).

Result is accurate

Next step: to retrain the model with more better quality images.

Attempt was made at moving the arm using remote procedure call by inputting the cartesian values.

The UR10 arm did not initially move to the intended position until adjustments were made to the Z value. 400mm needs to be added to the Z value before it actually reaches the intended position, which I am unable to figure out why.

Attempt was made to open and close the gripper, but by the end of the day, I still did not managed to in spite of studying the manual.

After adjustments were made to the model, there was slight improvement but still not enough. The Intel RealSense camera was able to detect the object, but sketchy at best. The background had to be plain and objects had to be moved around in order to be recognised. 

Week 04/08/2024 - 10/08/2024

 7/8/2024

Camera App

Sorted out camera connection issue. The Intel RealSense Viewer app was unable to detect camera. After testing out several cables, situation did not improve. The issue turned out to be the app itself. 

After uninstalling and reinstalling the app, the camera worked. 

Gripper Pads

Received new 3D-printed gripper pads:

Tested gripper pads with cylindrical object (marker pen):

Gripper pads were able to pick object up securely.

The test was then done on actual tools. Not all tools were cylindrical, so there was a possibility that the finger pads were not able to grip the tools securely.

Further tests showed that the gripper pads were able to securely grip the tools intended for this project. 

Debugging

Another attempt was made to write the programme to detect object via the Intel RealSense camera, but encountered this error each time:

Therefore, a step-by-step approach was taken to build the programme, ie write programme to test if::

• programme is able to access image file
• programme is able to access model
• programme is able to detect object in an image (outcome - failed)

The next step is to write a programme to check if programme is able to detect object in a saved video.

Week 28/07/2024 - 03/08/2024

This week was spent writing codes to rub object recognition for the 3 tools using the Intel RealSense D345 camera.

However, when I tried to run the codes on Jupyter, the kernel kept on dying. 

The same codes were also executed on Google Colab and VB without success. Debugging in process.