TSRPass Class Reference

Inherits from NSObject
Declared in TSRPass.h
TSRPass.mm

Overview

The TSRPass class provides methods for performing super-resolution rendering on input images using the Metal framework.

TSRPass is responsible for initializing the super-resolution rendering process with the specified input image size and super-resolution ratio. It also provides a render method that applies the super-resolution process to the input image and returns the processed image as an MTLTexture object.

Note:

  1. This class uses the Metal framework for performing super-resolution rendering and requires a device that supports Metal.
  2. All methods of TSRPass must be called within the same thread.
  3. The render:commandBuffer: and renderWithPixelBuffer: method should not be called before the initWithTSRAlgorithmType:device:inputWidth:inputHeight:srRatio:initStatusCode: method, and it should not be called after the deInit method.
  4. Initializing the TSRPass with TSRAlgorithmTypeProfessional is only available on iOS 15.0 or newer. If the iOS version is older, it will fallback to TSRAlgorithmTypeStandard.

Usage:

  1. Create a TSRPass instance using the initWithTSRAlgorithmType:device:inputWidth:inputHeight:srRatio:initStatusCode: method.
  2. Call the render:commandBuffer: method with the input image’s MTLTexture and an MTLCommandBuffer to perform super-resolution rendering.
  3. Obtain the processed image as an MTLTexture from the return value of the render:commandBuffer: method.
  4. deInit method should be called when the TSRPass object is no longer needed, to free up the memory and other resources.

– initWithTSRAlgorithmType:device:inputWidth:inputHeight:srRatio:initStatusCode:

Init TSRPass object for super-resolution.

- (instancetype)initWithTSRAlgorithmType:(TSRAlgorithmType)algorithmType device:(id<MTLDevice>)device inputWidth:(int32_t)inputWidth inputHeight:(int32_t)inputHeight srRatio:(float_t)srRatio initStatusCode:(TSRInitStatusCode *)initStatusCode

Parameters

algorithmType

The TSRAlgorithmType enum value representing the algorithm running mode to be used. It can be either STANDARD (fast mode) or PROFESSIONAL (quality mode). STANDARD mode prioritizes better performance in terms of speed, possibly sacrificing some result accuracy. PROFESSIONAL mode prioritizes result accuracy, even if the running speed is slower.
device

The MTLDevice which you are using.
inputWidth

The width of the input texture. This value should be between 8 to 4096.
inputHeight

The height of the input texture. This value should be between 8 to 4096.
srRatio

The magnification factor for super-resolution.
initStatusCode

A pointer to a TSRInitStatusCode enum value that will store the result of the initialization process. If initialization is successful, the value will be TSRInitStatusCodeSuccess. If initialization fails, the value will indicate the specific error that occurred.

Discussion

Init TSRPass object for super-resolution.

Declared In

TSRPass.h

– reInit:inputHeight:srRatio:

Reinitializes the TSRPass object for super-resolution rendering. The input parameters specify the new dimensions of the input image to be processed and the new magnification factor for super-resolution.

- (TSRInitStatusCode)reInit:(int32_t)inputWidth inputHeight:(int32_t)inputHeight srRatio:(float_t)srRatio

Parameters

inputWidth

The new width of the input image to be processed. This value should be between [8, 4096].
inputHeight

The new height of the input image to be processed. This value should be between [8, 4096].
srRatio

The new magnification factor for super-resolution. This value should be greater than zero and determines the scaling applied to the input image during the rendering process.

Return Value

TSRInitStatusCode enum value that will store the result of the reinitialization process. If reinitialization is successful, the value will be TSRInitStatusCodeSuccess. If reinitialization fails, the value will indicate the specific error that occurred.

Discussion

Reinitializes the TSRPass object for super-resolution rendering. The input parameters specify the new dimensions of the input image to be processed and the new magnification factor for super-resolution.

Declared In

TSRPass.h

– render:commandBuffer:

Performs the super-resolution rendering operation on the input image.
This method applies the super-resolution rendering process to the input image, improving its quality. The processed image is rendered onto a MTLTexture within the TSRPass object. And the return is the MTLTexture which has preformed super-resolution rendering .

- (id<MTLTexture>)render:(id<MTLTexture>)texture commandBuffer:(id<MTLCommandBuffer>)commandBuffer

Parameters

texture

The MTLTexture of the input image that needs to be processed for super-resolution.
commandBuffer

The MTLCommandBuffer which you are using.

Return Value

The MTLTexture that will be performed super-resolution rendering, which is stored inside the TSRPass object. The size of the output texture is (inputWidth * srRatio, inputHeight * srRatio).

Discussion

Performs the super-resolution rendering operation on the input image.
This method applies the super-resolution rendering process to the input image, improving its quality. The processed image is rendered onto a MTLTexture within the TSRPass object. And the return is the MTLTexture which has preformed super-resolution rendering .

Declared In

TSRPass.h

– renderWithPixelBuffer:

Performs the super-resolution rendering operation on the input pixel buffer.

- (CVPixelBufferRef)renderWithPixelBuffer:(CVPixelBufferRef)pixelBuffer

Parameters

pixelBuffer

The CVPixelBufferRef of the input image that needs to be processed for super-resolution. The width and height of the pixel buffer must match the inputWidth and inputHeight set during initialization, and the pixel format must be BGRA.

Return Value

The CVPixelBufferRef that has been performed super-resolution rendering. The size of the output pixel buffer is (inputWidth * srRatio, inputHeight * srRatio).

Discussion

Performs the super-resolution rendering operation on the input pixel buffer.

This method applies the super-resolution rendering process to the input pixel buffer, improving its quality. The processed pixel buffer is created and returned as a new CVPixelBufferRef.

Note: The SDK internally handles the lifecycle of the returned CVPixelBufferRef, therefore, the caller absolutely should not attempt to manually release the returned CVPixelBufferRef.

Declared In

TSRPass.h

– enableProSRAutoFallback:timeoutDurationMs:fallbackListener:

Configures and activates the automatic fallback mechanism for super-resolution processing.

- (void)enableProSRAutoFallback:(int)consecutiveTimeoutFrames timeoutDurationMs:(int)timeoutDurationMs fallbackListener:(FallbackListener)fallbackListener

Parameters

consecutiveTimeoutFrames

The initial threshold for the number of consecutive timeout frames before triggering a fallback. This value may be dynamically adjusted based on performance.
timeoutDurationMs

The duration in milliseconds that defines the timeout threshold for processing. If processing exceeds this duration, it is considered a timeout.
fallbackListener

The listener that will be notified when a fallback event occurs. The listener’s onFallback method will be called with the relevant parameters.

Discussion

Configures and activates the automatic fallback mechanism for super-resolution processing.

This method will determine a timeout when the processing time exceeds the specified timeoutDurationMs. If the number of consecutive timeout frames reaches the configured threshold consecutiveTimeoutFrames, an automatic fallback operation will be triggered.

The threshold for consecutive timeout frames, consecutiveTimeoutFrames, is dynamically adjusted based on the device’s performance. In cases of lower device performance, the threshold will be reduced. The adjustment mechanism is as follows:

  • If the processing time is twice timeoutDurationMs, the threshold will be halved.
  • If the processing time is three times timeoutDurationMs, the threshold will be reduced to one-third, and so on.

The minimum value for the threshold is 2. The formula for updating the threshold is: consecutiveTimeoutFrames = max(2, consecutiveTimeoutFrames / (timeConsume / timeoutDurationMs)), where timeConsume is the actual processing time.

Through this dynamic adjustment mechanism, the system can flexibly respond to the current performance conditions, ensuring stability and efficiency during the processing.

Declared In

TSRPass.h

– disableProSRAutoFallback

Disables the automatic fallback behavior for the super-resolution process. This method should be called to turn off the automatic fallback feature that was previously enabled using enableProSRAutoFallback:timeoutDurationMs:fallbackListener:. Once this method is invoked, the system will no longer trigger a fallback based on the configured parameters.

- (void)disableProSRAutoFallback

Discussion

Disables the automatic fallback behavior for the super-resolution process. This method should be called to turn off the automatic fallback feature that was previously enabled using enableProSRAutoFallback:timeoutDurationMs:fallbackListener:. Once this method is invoked, the system will no longer trigger a fallback based on the configured parameters.

Declared In

TSRPass.h

– benchmarkProSR:inputHeight:srRatio:

Evaluates the rendering time consumption of the PROFESSIONAL algorithm. This method assesses the execution time in milliseconds for the PROFESSIONAL algorithm based on the given input dimensions. This method should not be called on the main thread, as it may take approximately 2 to 5 seconds to complete.

- (int)benchmarkProSR:(int)inputWidth inputHeight:(int)inputHeight srRatio:(float)srRatio

Parameters

inputWidth

The width of the input.
inputHeight

The height of the input.
srRatio

The super-resolution ratio.

Return Value

The estimated rendering time consumption in milliseconds, or -1 if the execution fails.

Discussion

Evaluates the rendering time consumption of the PROFESSIONAL algorithm. This method assesses the execution time in milliseconds for the PROFESSIONAL algorithm based on the given input dimensions. This method should not be called on the main thread, as it may take approximately 2 to 5 seconds to complete.

This method only takes effect if the TSRAlgorithmType used to init the TSRPass is not set to TSRAlgorithmTypeStandard. If the algorithm type is STANDARD, the evaluation may not be applicable. If the execution of the algorithm fails for any reason, this method will return -1.

Declared In

TSRPass.h

– forceProSRFallback:

Switches between the PROFESSIONAL and STANDARD algorithms. This method enables or disables the use of the STANDARD algorithm. When enable is true, the system will switch to the STANDARD algorithm; otherwise, it will use the PROFESSIONAL algorithm. This method only takes effect if the TSRAlgorithmType used to init the TSRPass is not set to TSRAlgorithmTypeStandard.

- (void)forceProSRFallback:(bool)enable

Parameters

enable

true to switch to the STANDARD algorithm; false to use the PROFESSIONAL algorithm.

Discussion

Switches between the PROFESSIONAL and STANDARD algorithms. This method enables or disables the use of the STANDARD algorithm. When enable is true, the system will switch to the STANDARD algorithm; otherwise, it will use the PROFESSIONAL algorithm. This method only takes effect if the TSRAlgorithmType used to init the TSRPass is not set to TSRAlgorithmTypeStandard.

Declared In

TSRPass.h

– setParametersWithBrightness:saturation:contrast:sharpness:

Adjusts the parameters of the rendering operator. It needs to take effect after TSRPass#initWithDevice is called.

- (void)setParametersWithBrightness:(float)brightness saturation:(float)saturation contrast:(float)contrast sharpness:(float)sharpness

Parameters

brightness

The brightness level to apply. Valid values are between 0 and 100, with a default value of 50.
saturation

The saturation level to apply. Valid values are between 0 and 100, with a default value of 50.
contrast

The contrast level to apply. Valid values are between 0 and 100, with a default value of 50.
sharpness

The sharpness level to apply. Valid values are between 0 and 100, with a default value of 0.

Discussion

Adjusts the parameters of the rendering operator. It needs to take effect after TSRPass#initWithDevice is called.

Declared In

TSRPass.h

– deInit

Releases the resources.
This method should be called when the TSRPass object is no longer needed, to free up the memory and other resources.

- (void)deInit

Discussion

Releases the resources.
This method should be called when the TSRPass object is no longer needed, to free up the memory and other resources.

Declared In

TSRPass.h