| | |
|---|
| | | |
|---|
| | | 5. If you have GPU with Tensor Cores (nVidia Titan V / Tesla V100 / DGX-2 and later) speedup Detection 3x, Training 2x: |
|---|
| | | `\darknet.sln` -> (right click on project) -> properties -> C/C++ -> Preprocessor -> Preprocessor Definitions, and add here: `CUDNN_HALF;` |
|---|
| | | |
|---|
| | | **Note:** CUDA must be installed only after that MSVS2015 had been installed. |
|---|
| | | |
|---|
| | | ### How to compile (custom): |
|---|
| | | |
|---|
| | |
|---|
| | | |
|---|
| | | More information about training by the link: http://pjreddie.com/darknet/yolo/#train-voc |
|---|
| | | |
|---|
| | | **Note:** If during training you see `nan` values in some lines then training goes well, but if `nan` are in all lines then training goes wrong. |
|---|
| | | **Note:** If during training you see `nan` values for `avg` (loss) field - then training goes wrong, but if `nan` is in some other lines - then training goes well. |
|---|
| | | |
|---|
| | | ## How to train with multi-GPU: |
|---|
| | | |
|---|
| | |
|---|
| | | |
|---|
| | | * Also you can get result earlier than all 45000 iterations. |
|---|
| | | |
|---|
| | | **Note:** If during training you see `nan` values in some lines then training goes well, but if `nan` are in all lines then training goes wrong. |
|---|
| | | **Note:** If during training you see `nan` values for `avg` (loss) field - then training goes wrong, but if `nan` is in some other lines - then training goes well. |
|---|
| | | |
|---|
| | | ### How to train tiny-yolo (to detect your custom objects): |
|---|
| | | |
|---|
| | | Do all the same steps as for the full yolo model as described above. With the exception of: |
|---|
| | | * Download default weights file for yolov2-tiny-voc: http://pjreddie.com/media/files/yolov2-tiny-voc.weights |
|---|
| | | * Get pre-trained weights yolov2-tiny-voc.conv.13 using command: `darknet.exe partial cfg/yolov2-tiny-voc.cfg yolov2-tiny-voc.weights yolov2-tiny-voc.conv.13 13` |
|---|
| | | * Make your custom model `yolov2-tiny-obj.cfg` based on `cfg/yolov2-tiny-voc.cfg` instead of `yolov3.cfg` |
|---|
| | | * Start training: `darknet.exe detector train data/obj.data yolov2-tiny-obj.cfg yolov2-tiny-voc.conv.13` |
|---|
| | | * Download default weights file for yolov3-tiny: https://pjreddie.com/media/files/yolov3-tiny.weights |
|---|
| | | * Get pre-trained weights `yolov3-tiny.conv.15` using command: `darknet.exe partial cfg/yolov3-tiny.cfg yolov3-tiny.weights yolov3-tiny.conv.15 15` |
|---|
| | | * Make your custom model `yolov3-tiny-obj.cfg` based on `cfg/yolov3-tiny_obj.cfg` instead of `yolov3.cfg` |
|---|
| | | * Start training: `darknet.exe detector train data/obj.data yolov3-tiny-obj.cfg yolov3-tiny.conv.15` |
|---|
| | | |
|---|
| | | For training Yolo based on other models ([DenseNet201-Yolo](https://github.com/AlexeyAB/darknet/blob/master/build/darknet/x64/densenet201_yolo.cfg) or [ResNet50-Yolo](https://github.com/AlexeyAB/darknet/blob/master/build/darknet/x64/resnet50_yolo.cfg)), you can download and get pre-trained weights as showed in this file: https://github.com/AlexeyAB/darknet/blob/master/build/darknet/x64/partial.cmd |
|---|
| | | If you made you custom model that isn't based on other models, then you can train it without pre-trained weights, then will be used random initial weights. |
|---|
| | |
|---|
| | | |
|---|
| | | 2. Once training is stopped, you should take some of last `.weights`-files from `darknet\build\darknet\x64\backup` and choose the best of them: |
|---|
| | | |
|---|
| | | For example, you stopped training after 9000 iterations, but the best result can give one of previous weights (7000, 8000, 9000). It can happen due to overfitting. **Overfitting** - is case when you can detect objects on images from training-dataset, but can't detect ojbects on any others images. You should get weights from **Early Stopping Point**: |
|---|
| | | For example, you stopped training after 9000 iterations, but the best result can give one of previous weights (7000, 8000, 9000). It can happen due to overfitting. **Overfitting** - is case when you can detect objects on images from training-dataset, but can't detect objects on any others images. You should get weights from **Early Stopping Point**: |
|---|
| | | |
|---|
| | |  |
|---|
| | | |
|---|
| | |
|---|
| | | `darknet.exe detector calc_anchors data/obj.data -num_of_clusters 9 -width 416 -height 416` |
|---|
| | | then set the same 9 `anchors` in each of 3 `[yolo]`-layers in your cfg-file |
|---|
| | | |
|---|
| | | * check that each object are mandatory labeled in your dataset - no one object in your data set should not be without label. In the most training issues - there are wrong labels in your dataset (got labels by using some conversion script, marked with a third-party tool, ...). Always check your dataset by using: https://github.com/AlexeyAB/Yolo_mark |
|---|
| | | |
|---|
| | | * desirable that your training dataset include images with objects at diffrent: scales, rotations, lightings, from different sides, on different backgrounds |
|---|
| | | |
|---|
| | | * desirable that your training dataset include images with non-labeled objects that you do not want to detect - negative samples without bounded box (empty `.txt` files) |
|---|
| | | |
|---|
| | | * for training with a large number of objects in each image, add the parameter `max=200` or higher value in the last layer [region] in your cfg-file |
|---|
| | | |
|---|
| | | * for training for small objects - set `layers = -1, 11` instead of https://github.com/AlexeyAB/darknet/blob/6390a5a2ab61a0bdf6f1a9a6b4a739c16b36e0d7/cfg/yolov3.cfg#L720 |
|---|
| | | and set `stride=4` instead of https://github.com/AlexeyAB/darknet/blob/6390a5a2ab61a0bdf6f1a9a6b4a739c16b36e0d7/cfg/yolov3.cfg#L717 |
|---|
| | | |
|---|
| | | * General rule - you should keep relative size of objects in the Training and Testing datasets roughly the same: |
|---|
| | | |
|---|
| | | * `train_network_width * train_obj_width / train_image_width ~= detection_network_width * detection_obj_width / detection_image_width` |
|---|
| | | * `train_network_height * train_obj_height / train_image_height ~= detection_network_height * detection_obj_height / detection_image_height` |
|---|
| | | |
|---|
| | | * to speedup training (with decreasing detection accuracy) do Fine-Tuning instead of Transfer-Learning, set param `stopbackward=1` in one of the penultimate convolutional layers before the 1-st `[yolo]`-layer, for example here: https://github.com/AlexeyAB/darknet/blob/0039fd26786ab5f71d5af725fc18b3f521e7acfd/cfg/yolov3.cfg#L598 |
|---|
| | | |
|---|
| | | 2. After training - for detection: |
|---|
