Agriculture commodities are commodities that have a high economic worth and the potential to be developed further. The green and red apple, in instance, is one type of fruit that has the potential to be cultivated as part of agriculture. The apple economy is reasonably steady, particularly with regard to the supply of production to the market. The purpose of this research is to enhance the performance of the CNN-based model and make it capable of precise detection of the green and red apple fruitlet. To enhance the overall performance of the model, the revised CNN-based YOLOv5 ensemble model was implemented with the SiLU (Sigmoid Linear Units activation function), Batch Normalization, and SGD (Stochastic Gradient Descent) algorithms. The combination of activation function, optimization, batch normalization, and ensemble technique can be later used to enhance the YOLOv5 ensemble model and used to detect the green and red apple fruitlet with the benefits of utilizing limited resources. This is possible thanks to the combination of the activation function, optimization, batch normalization, and ensemble technique. According to the findings of the comprehensive research, the accuracy of the updated yolo ensemble model has climbed into 97.8%, 92.1%, 95% percent of accuracy mAP for green, red and both apples together compared to previous model.