* fix optimizer wrapper counts

* fix ut

* fix amp

* fix amp

* make logic consistent with torch.autocast

* support multiple device

* fix as comment

* fix as comment

* avoid circle import

* [Enhance] Support scheduling betas with MomentumScheduler.

* enhance ut

* test adam betas

* enhance ut

* enhance ut

* fix build multiple runner error

* fix comments

* fix cpu ci

* Fix ema hook and add unit test

* save state_dict of ema.module

save state_dict of ema.module

* replace warning.warn with MMLogger.warn

* fix as comment

* fix bug

* fix bug

* support dynamic interval in iterbasedtrainloop

* update typehint

* update typehint

* add dynamic interval in epochbasedtrainloop

* update

* fix

Co-authored-by: luochunhua.vendor <luochunhua@pjlab.org.cn>

* fix bug of not save-best in iteration-based training

* revise the unit test

* fix unit test of autocast

* fix compatiblity of unit test of optimizerwrapper

* clean code

* fix as comment

* fix docstring

* assert image shape before forward

* add unit test

* enhance error message

* allow gray image input

* fix as comment

* fix unit test

* fix unit test

* impl lr and momentum visualizer

* provide fakerun

* refine data_root and data_prefix params

* modify unittest

* [Feature]: Set different seed for diff rank

* [Feature]: Add log

* [Fix]: Fix lint

* [Fix]: Fix docstring

* [Fix]: Fix sampler seed

* [Fix]: Fix log bug

* [Fix]: Change diff_seed to diff_rank_seed

* [Fix]: Fix lint

* [Feat] Support mps

* fix docstring

* Fix docstring format of BaseDataElement

* fix docstring

* fix RuntimeError of SyncBuffersHook

* add UT

* [Feat] Support revert syncbn

* use logger.info but not warning

* fix info string

* add autocast wrapper

* fix docstring

* fix docstring

* fix compare version

* fix unit test

* fix incompatible arguments

* fix as comment

* fix unit test

* rename auto_cast to autocast

* [Feat] Support save best ckpt

* reformat code

* rename function and reformat code

* fix logging info

* LogProcessor support custom significant digit

* rename to num_digits

* modify cuda() to to()

* rollback load_checkpoint

* refine runner

* add TODO

* [Refactor]:Refactor `after_val_epoch` to make it output metric by epoch

* add an option for user to choose the way of outputing metric

* rename variable

* reformat docstring

* add type alias

* reformat code

* add test function

* add comment and test code

* add comment and test code

* [Feat] Support FSDP Training

* fix version comparison

* change param format and move `FSDP_WRAP_POLICY` to wrapper file

* add docstring and type hint,reformat code

* fix type hint

* fix typo, reformat code

* move import resource

* move import resource

* [Enhance] dump messagehub in runner.resume

* delete unnecessary code

* delete debugging code

Co-authored-by: imabackstabber <312276423@qq.com>

* support resume from ceph

* move func and refine

* delete symlink

* fix unittest

* perserve _allow_symlink and symlink