21/7/2024
Collected my objects / tools from the university.
Proceeded to take pictures to build the custom dataset.
21/7/2024
Proceeded to annotate and label pictures using CVAT (apps.cvat.ai)
22/7/2024 - 23/7/2024
Started training the model after annotation using Google Colab. Took more pictures for validation.
14/7/2024
Decided to start working on the codes for object detection using Intel RealSense D435 and YOLOv8.
3 objects have been chosen for the object detection:
This week, I continued writing literature summary for the following papers:
Objective/Aim:
Develop a mobile grasping system for robotic arm.
Method:
First, the YOLOv4 algorithm is introduced to identify and locate the individual objects. Secondly, the output frame can be used as an input of the GrabCut algorithm to segment the objects from the background for calculating the capture orientation.
Thirdly, the conversion between the coordinate systems is conducted to acquire the positions of objects, and the obtained information is sent to the robotic arm to complete the grasping task on the Robot Operating System (ROS).
Process flow:
Robot performs SLAM through lidar to reach the desired position.
System will turn on the camera to identify and locate the objects, and compute the pose of each object.
User enters the object category to be captured, and the system will make a judgment.
Computer System/Software:
The system is implemented in Ubuntu16.04 on a Lenovo 2.6 GHz processor RTX 2060 with 16 GB RAM and 8 GB video memory
Setup:
The robot is equipped with a Hokuyo lidar, RealSense D435 camera.
Evaluation:
Experiment is done on 7 objects of different colour and shapes placed in the scene at different positions and angles. 3 measurements are obtained: grasping times, successful times and success rate.
Conclusion:
The results show that the proposed method can complete the robot’s navigation to the target position to realize the grasping function of the specified object.
Future works/considerations:
Increase the accuracy of the system by improving the moving object extraction algorithm.
Title: Robotic Arm Grasping and Placing Using Edge Visual Detection System
Link: https://ieeexplore.ieee.org/document/5899202
Objective/Aim:
Aims to develop the object visional detection system that can be applied to the robotic arm grasping and placing.
Method:
Accurately measure the relative distance between the object and robot arm using the edge detection algorithm with a camera device.
Uses image processing and shape matching to identify object
Placement of object is done using the same image processing method.
Computer System/Software:
Borland C++ Builder 6 programming was used for the image processing
Setup:
The robotic arm in the study uses seven RX-64 servomotors and line moving rail.
The webcam with the resolution of 320 × 240(pixel) is installed at the upper location of arm clip.
Evaluation:
Grabbing experiments using four kinds of color (red, yellow, green, and blue) ball for
each one, each two, and each three, respectively, indoor and the hall under different environment.
Conclusion:
Accomplished three tasks, i.e., “find the ball”, “clip the ball” and “put the ball”.
More importantly, this system is a really streamlined and inexpensive system.
Future works/considerations:
Focus on finding, grasping and placement of objects in different shapes and colors.
Title: ReorientBot: Learning Object Reorientation for Specific-Posed Placement
Link: https://ieeexplore.ieee.org/abstract/document/9811881
Objective/Aim:
To present a robotic system that can rearrange objects to a specific goal state, including reorientation and regrasping for final placement.
Method:
Systems runs detection, post estimation and motion planning to rearrange objects.
i) 6D pose estimation and volumetric reconstruction
ii) Motion waypoint selection that pairs start and end waypoints via learned filtering
iii) Trajectory generation by motion planning using the selected waypoints.
Computer System/Software:
ROS, PyTorch to implement the learned models. PyBullet as physics engine to simulate behaviour of objects.
Setup:
Franka Emika Panda robot, Realsense D435 mounted on robotic arm.
Two types of suction gripper:
I-shape
L-shape
Evaluation:
Done in both simulation and real-world. 6 large/medium sized objects were used in both simulation and real-world experiments: drill, cracker box, sugar box, mustard bottle, pitcher and detergent.
Conclusion:
The authors’ system improves in both efficiency and success rate and has shown capable, dynamic reorientation for significant rotation and precise placement in various target configurations.
Future works/considerations:
Possible combinations of learning models with traditional motion planning.
Initial objective and goals:
Objective
By the end of this project, I would have investigated if a robot system is able to pick an object based on spoken instruction through vision and then place the item picked onto the speaker’s palm within 15 seconds.
Goals:
Investigate if:
Robot arm is able to identify object based on speech and vision
Robot is able to automatically grasp objects through vision.
Robot is able to grasp objects of different shapes and sizes
Robot is able to grasp an object in a different orientation.
Place the object onto the palm of a person.
Objective
By the end of this project, I would have investigated if a robot system is able to pick an object through vision and then place the item picked onto a person's palm within 15 seconds.
Goals:
Investigate if:
Robot arm is able to identify object based on vision
Robot is able to automatically grasp objects through vision.
Robot is able to grasp an object in a different orientation.
Place the object onto the palm of a person.
Literature Summary
Read through 3 papers and wrote a summary for the 3 papers:
Title: A vision-based robot grasping system
Link: https://ieeexplore.ieee.org/abstract/document/9745523Objective/Aim:
Increasing the grasp pose detection accuracy for a variety of daily household objects only from the visual sensing
Setup:
Franka Panda robot arm equipped with parallel gripper. An intel RealSense D435 is attached to the arm, just above the gripper.
Method:
For the grasp detector, the system uses a densely connected Feature Pyramid Network (FPN) feature extractor.
For the robot system, the vision measurement algorithm generates the grasp pose from a single input modality directly, the depth image.
Computer system/software:
Tensorflow deep learning library, written in Python. PC running Ubuntu 18.04, equipped with NVidia GeForce 1080 Ti GPU Intel Core i7-6700K CPU @ 4.00GHz x 8 processor and 32G memory.
Evaluation:
Two public datasets were used for validation: Cornell Grasp Dataset and Jacquard Dataset.
Three types of experiments were done: i) Grasping a single object at different poses; ii) grasping 51 different objects that are not included in the public datasets; grasping multiple objects at one time.
Conclusion:
The three experiments done proved that the model is able to grasp all kinds of daily objects in various poses.
Future works/considerations:
Incorporate tactile sensing in the grasping system to give a higher grasping success rate.
Incorporate RGB-D visual servoing controller into the parallel gripper grasp system to eliminate execution error.
Title: Vision-Based Robotic Arm Control Algorithm Using Deep Reinforcement Learning for Autonomous Objects Grasping
Link: https://www.mdpi.com/2076-3417/11/17/7917Objective/Aim:
To evolve the grasping task by reaching the intended object based on deep reinforcement learning.
To compute the robot arm kinematics to grasp a specific object.
Setup:
5-DOF robotic arm equipped with an 8MP resolution camera.
Method:
Use YOLOv5 for object detection and position
Use backward projection for extraction of the object’s 3D position
Use inverse kinematics to compute the angles of the joints at the detected position\
Employ Deep Deterministic Policy algorithm to teach the arm to autonomously reach the wanted object
Computer System/Software:
Intel Core i7 8th generation processor CPU, 16GB RAM
GPU - NVIDIA GeForce GTX 960
Ubuntu operating system
Anaconda Python package: Jupyter, Tensorflow, Keras and Matplotlib
Evaluation:
Evaluate the efficiency of the 5-DOF arm robot to grasp a determined object.
Train the model for 400 episodes, obtain accuracy and error results.
Conclusion:
Despite some error, every joint angle can be calculated and the end-effector can reach the determined location.
Decrease in error range throughout episodes proved that the reinforcement learning algorithm can reach a targeted object with an inverse kinematic of the robot arm.
Future works/considerations:
Expand model by integrating the pick and place task.
Title: Robotic Object Recognition and Grasping with Natural Background
Link: https://journals.sagepub.com/doi/full/10.1177/1729881420921102Objective/Aim:
Introduce a novel, efficient grasp synthesis method that can be ued for closed-loop robotic grasping.
Setup:
Computer connected to robot control cabinet
6-DOF SD700E Industrial Robot Arm with EFG20 electric gripper
Logitech C310 camera mounted on the end flange of the arm
Method:
Edge detection, superpixel segmentation, shape matching
Using relative distance between object centroid and the gripper, the algorithm guides the robot to move the gripper to the object and form a proper grabbing posture to complete the task.
Computer System/Software:
-
Evaluation:
Changing the state of the table
Changing the relative order of objects
Changing the postures and positions
Conclusion: -
Future works/considerations: -
11/10/2024 Submitted report, codes and demo video: Updated Gantt Chart: Reflection: When it comes to technical research and experiments, so...
